feedwatcher

Documentation

Index

• Variables
• type ErrMailDeliveryFailed
  • func (e *ErrMailDeliveryFailed) Error() string
• type FeedCrawlRequest
• type FeedCrawlResponse
• type FeedWatcher
  • func NewFeedWatcher(feedInfo db.FeedInfo, crawlChan chan *FeedCrawlRequest, ...) *FeedWatcher
  • func (fw *FeedWatcher) CrawlFeed() (r *FeedCrawlResponse)
  • func (fw *FeedWatcher) Crawling() (r bool)
  • func (fw *FeedWatcher) LoadGuidsFromDb(max int) (map[string]bool, error)
  • func (fw *FeedWatcher) PollFeed() bool
  • func (fw *FeedWatcher) Polling() bool
  • func (fw *FeedWatcher) StopPoll()
  • func (fw *FeedWatcher) UpdateFeed(resp *FeedCrawlResponse) error

Variables

var After = func(d time.Duration) <-chan time.Time {
	return time.After(d)
}

After allows for stubbing out in test

var ErrAlreadyCrawlingFeed = errors.New("already crawling feed")

ErrAlreadyCrawlingFeed is when you are already Crawling a feed

var ErrCrawlerNotAvailable = errors.New("no crawler available")

ErrCrawlerNotAvailable is when the crawler channel would block

Types

type ErrMailDeliveryFailed

type ErrMailDeliveryFailed struct {
	// contains filtered or unexported fields
}

ErrMailDeliveryFailed captures the specific reason for a mail failure

func (*ErrMailDeliveryFailed) Error

func (e *ErrMailDeliveryFailed) Error() string

type FeedCrawlRequest

type FeedCrawlRequest struct {
	URI          string
	ResponseChan chan *FeedCrawlResponse
}

FeedCrawlRequest is how to request a URL to be crawled by a crawler instance.

type FeedCrawlResponse

type FeedCrawlResponse struct {
	URI                    string
	Body                   []byte
	Feed                   *gofeed.Feed
	Items                  []*gofeed.Item
	HTTPResponseStatus     string
	HTTPResponseStatusCode int
	Error                  error
}

FeedCrawlResponse is we get back from the crawler.

Body, Feed, Items and HTTPResponseStatus are only guaranteed to be non zero value if Error is non nil.

type FeedWatcher

type FeedWatcher struct {
	FeedInfo db.FeedInfo

	GUIDCache         map[string]bool
	LastCrawlResponse *FeedCrawlResponse
	After             func(d time.Duration) <-chan time.Time // Allow for mocking out in test.
	Logger            logrus.FieldLogger
	// contains filtered or unexported fields
}

FeedWatcher controlls the crawling of a Feed. It keeps state about the GUIDs it's seen, sends crawl requests and deals with responses.

func NewFeedWatcher

func NewFeedWatcher(
	feedInfo db.FeedInfo,
	crawlChan chan *FeedCrawlRequest,
	mailChan chan *mail.Request,
	dbh db.Service,
	GUIDCache []string,
	minSleep int64,
	maxSleep int64,
) *FeedWatcher

NewFeedWatcher returns a new FeedWatcher instance.

func (*FeedWatcher) CrawlFeed

func (fw *FeedWatcher) CrawlFeed() (r *FeedCrawlResponse)

CrawlFeed will send a request to crawl a feed over it's crawl channel

func (*FeedWatcher) Crawling

func (fw *FeedWatcher) Crawling() (r bool)

Crawling returns true if the FeedWatcher is crawling a feed (actually waiting for a response from the crawler).

func (*FeedWatcher) LoadGuidsFromDb

func (fw *FeedWatcher) LoadGuidsFromDb(max int) (map[string]bool, error)

LoadGuidsFromDb populates the internal cache of GUIDs by getting the most recent GUIDs it knows about. To retrieve all GUIDs set max to -1.

func (*FeedWatcher) PollFeed

func (fw *FeedWatcher) PollFeed() bool

PollFeed is designed to be called as a Goroutine. Use UpdateFeed to just update once.

func (*FeedWatcher) Polling

func (fw *FeedWatcher) Polling() bool

Polling returns true if the FeedWatcher is Polling a feed.

func (*FeedWatcher) StopPoll

func (fw *FeedWatcher) StopPoll()

StopPoll will cause the PollFeed loop to exit.

func (*FeedWatcher) UpdateFeed

func (fw *FeedWatcher) UpdateFeed(resp *FeedCrawlResponse) error

UpdateFeed will crawl the feed, check for new items, mail them out and update the database with the new information