dupefilter

Documentation

Index

• func RFPDupeFilterMiddleware() antch.Middleware
• type RFPDupeFilter
  • func (f *RFPDupeFilter) Send(req *http.Request) (*http.Response, error)

Functions

func RFPDupeFilterMiddleware

func RFPDupeFilterMiddleware() antch.Middleware

Types

type RFPDupeFilter

type RFPDupeFilter struct {
	// IncludeHeaders specifies what header value is included to
	// fingerprint compute.
	IncludeHeaders []string
	// contains filtered or unexported fields
}

RFPDupeFilter is a middleware of HTTP downloader to filter duplicate requests, based on request fingerprint.

func (*RFPDupeFilter) Send

func (f *RFPDupeFilter) Send(req *http.Request) (*http.Response, error)