scraper

Documentation

Index

• Constants
• func Close(c io.Closer)
• type DownloadState
• type Options
• type Scraper
  • func NewScraper(options Options) Scraper
  • func (s Scraper) Start()

Constants

const (
	DOWNLOADING DownloadState = iota
	SUCCESS                   = iota
	SKIPPED                   = iota
	FAILED                    = iota
)

Functions

func Close

func Close(c io.Closer)

Close is designed to handle a defer closed on a closer. Correctly and fatally exiting if a error occurs on the close.

Types

type DownloadState

type DownloadState int

DownloadState is the outcome constant of the download process. Used to determine the message to be generated and shown to the user.

type Options

type Options struct {
	//  The directory in which we will be downloading all the images into, based on the folder name
	//  of the given sub-reddit.
	OutputDirectory string
	// The total number of images to download max per sub-reddit before we continue to the next one.
	ImageLimit int
	// If set to true, the tool will scrape the front page of reddit for the current most
	// active sub-reddits and then scrape all the links directly from them sub-reddits.
	FrontPage bool
	// if the images are being downloaded directly into the root folder and nothing else.
	RootFolderOnly bool
	// You can change this to adjust on what kind of images you get from Reddits filtering
	// options (hot, new, rising, controversial, top), hot is the default by reddit while
	// also the default in the tool.
	PageType string
	// What subreddits are going to be scrapped for downloading of sad images. If front page is
	// parsed as true then the front page will be pushed onto the sub reddit listings.
	Subreddits []string
	// If the loading progress bar should be displayed or not. Simply used for headless progressing
	// or testing that helps with minimising the amount of output that is generated to the console.
	DisplayLoading bool
}

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper is the type that will be containing all the configuration and data used for the parsing process. Including references to already downloaded ids + channels for the message and image pump.

func NewScraper

func NewScraper(options Options) Scraper

NewRedditScraper creates a instance of the reddit reddit used for taking images from the reddit site and downloading them into the given directory. Additionally sets the default options and data into the reddit reddit.

func (Scraper) Start

func (s Scraper) Start()

Start is exposed and called into when a new Scraper is created, this is called when the cli commands are parsed and the application is ready to start.