scraper

Documentation

Overview

Package scraper provides video file scraping functionality

Index

• Constants
• Variables
• func ScrapeFFProbe(filename string, vid *models.Video, logger *logrus.Entry) error
• func ScrapeSHA512(filename string, vid *models.Video, logger *logrus.Entry) error
• type FFFormatInfo
• type FFProbeData
  • func (d *FFProbeData) GetFirstSteamByType(t string) *FFStreamInfo
• type FFStreamInfo
• type FieldIndexMap
• type NameScrapingPreset
• type Scrape
  • func (scr Scrape) Stop()
  • func (scr Scrape) String() string
• type ScrapeStatus
  • func (s ScrapeStatus) MarshalJSON() ([]byte, error)
  • func (s ScrapeStatus) String() string
• type Scraper
  • func New(vRepo repos.VideoRepo, functions []ScrapingFunc, logger *logrus.Entry) *Scraper
  • func NewDefault(vRepo repos.VideoRepo, logger *logrus.Entry) *Scraper
  • func (s *Scraper) Start(rootDir string) error
  • func (s *Scraper) Status(rootDir string) *Scrape
  • func (s *Scraper) StatusAll() []Scrape
  • func (s *Scraper) Stop(rootDir string)
  • func (s *Scraper) StopAll()
• type ScrapingFunc
  • func MakeFileNameScraper(presetName string) (ScrapingFunc, error)
  • func MustMakeFileNameScraper(presetName string) ScrapingFunc

Constants

const (
	// FldTitle marks a match for the "Title"" field of a video
	FldTitle = "Title"
	// FldArtist marks a match for the "Artist"" field of a video
	FldArtist = "Artist"
	// FldRelatedMedium marks a match for the "RelatedMedium"" field of a video
	FldRelatedMedium = "RelatedMedium"
	// FldMediumDetail marks a match for the "MediumDetail"" field of a video
	FldMediumDetail = "MediumDetail"
	// FldDescription marks a match for the "Description"" field of a video
	FldDescription = "Description"
	// FldLanguage marks a match for the "Language"" field of a video
	FldLanguage = "Language"
	// FldIdentifier marks a match for the "Identifier" field of a video
	FldIdentifier = "Identifier"
	// StatusQueued is the status a scrape has when it waits to be started
	StatusQueued = iota
	// StatusRunning is the status of a scrape that is currently active
	StatusRunning
	// StatusFinished is the status of a scrape that has been finished successfully
	StatusFinished
	// StatusFailed is the status of a scrape that has failed for some reason. To look up the reason, see the Err field
	// of the scrape struct
	StatusFailed
	// StatusCancelled is the status of a scrape that has been cancelled by the user
	StatusCancelled
)

Variables

var (

	// ErrAlreadyQueued is the error that is returned when scraping the same or a parent directory is already inside
	// the scraping queue
	ErrAlreadyQueued = fmt.Errorf("A scraping operation is already queued for this directory")
)

Functions

func ScrapeFFProbe

func ScrapeFFProbe(filename string, vid *models.Video, logger *logrus.Entry) error

ScrapeFFProbe uses the ffprobe commandline tool to scrape the video metadata from its JSON output

func ScrapeSHA512

func ScrapeSHA512(filename string, vid *models.Video, logger *logrus.Entry) error

ScrapeSHA512 calculates the SHA-512 sum of the video file and adds it to the video metadata provided

Types

type FFFormatInfo

type FFFormatInfo struct {
	Filename       string            `json:"filename"`
	NumStreams     uint              `json:"nb_streams"`
	NumPrograms    uint              `json:"nb_programs"`
	FormatName     string            `json:"format_name"`
	FormatLongName string            `json:"format_long_name"`
	StartTime      string            `json:"start_time"`
	Duration       string            `json:"duration"`
	Size           string            `json:"size"`
	Bitrate        string            `json:"bit_rate"`
	ProbeScore     uint              `json:"probe_score"`
	Tags           map[string]string `json:"tags"`
}

FFFormatInfo contains information about the media format.

type FFProbeData

type FFProbeData struct {
	Format  *FFFormatInfo   `json:"format"`
	Streams []*FFStreamInfo `json:"streams"`
}

FFProbeData is the data format the tool ffprobe generates in JSON

func (*FFProbeData) GetFirstSteamByType

func (d *FFProbeData) GetFirstSteamByType(t string) *FFStreamInfo

GetFirstSteamByType returns the first stream in the media file's streams that has the given type

type FFStreamInfo

type FFStreamInfo struct {
	CodecType     string `json:"codec_type"`
	CodecName     string `json:"codec_name"`
	CodecLongName string `json:"codec_long_name"`
	Width         int    `json:"width"`
	Height        int    `json:"height"`
	Bitrate       string `json:"bit_rate"`
}

FFStreamInfo contains information about a stream inside a media file This struct does not contain all of the fields returned by ffprobe

type FieldIndexMap

type FieldIndexMap map[string]int

FieldIndexMap describes the correlation between a field of a video struct and the index in the scraping result that will be used to fill this field. It is used in the "FileNameSchema" scraper for configuring which field will be filled with the result of which capture group of the regular expression

type NameScrapingPreset

type NameScrapingPreset struct {
	Name     string
	Regex    string
	FieldMap FieldIndexMap
}

NameScrapingPreset defines a named preset for scraping file names It is used to make it easier for repeating scrapes of a special kind

type Scrape

type Scrape struct {
	// The current status of the scrape. See the Status* constants for possible values
	Status ScrapeStatus `json:"status"`
	// The root directory where the scraping started - since multiple scrapes in the same directory are not allowed,
	// this can also be seen as a unique ID
	RootDir string `json:"rootDir"`
	// The directory currently being traversed
	CurrentDir string `json:"currentDir"`
	// The file currently being scanned
	CurrentFile string `json:"currentFile"`
	// The number of video files already scraped
	NumFiles uint `json:"filesScraped"`
	// The number of new video files scraped
	NumNewFiles uint `json:"newFiles"`
	// The number of already existing video files updated
	NumUpdatedFiles uint `json:"updatedFiles"`
	// The time the scape has started
	StartedAt time.Time `json:"startedAt"`
	// If the scrape has failed, this is the error that caused it
	Err error `json:"error"`
	// contains filtered or unexported fields
}

Scrape describes a video scraping operation currently running

func (Scrape) Stop

func (scr Scrape) Stop()

Stop sends the stop signal to the current scrape's goroutine This method blocks until the scrape has stopped

func (Scrape) String

func (scr Scrape) String() string

type ScrapeStatus

type ScrapeStatus uint

ScrapeStatus defines the status of a scrape

func (ScrapeStatus) MarshalJSON

func (s ScrapeStatus) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.marshaler interface returning the name of the status

func (ScrapeStatus) String

func (s ScrapeStatus) String() string

Converts the scrape status into a readable name

type Scraper

type Scraper struct {
	// contains filtered or unexported fields
}

A Scraper runs a set of Scraping functions on files fed to it

func New

func New(vRepo repos.VideoRepo, functions []ScrapingFunc, logger *logrus.Entry) *Scraper

New returns a new scraper with the given functions set as scraping functions

func NewDefault

func NewDefault(vRepo repos.VideoRepo, logger *logrus.Entry) *Scraper

NewDefault creates a new scraper that is setup using the default scraping functions

func (*Scraper) Start

func (s *Scraper) Start(rootDir string) error

Start begins scraping from the given root directory. It returns the ID of the

func (*Scraper) Status

func (s *Scraper) Status(rootDir string) *Scrape

Status retrieves the status of the scrape running for the given root directory

func (*Scraper) StatusAll

func (s *Scraper) StatusAll() []Scrape

StatusAll returns the status object for all scrapes currently available in the scrape list

func (*Scraper) Stop

func (s *Scraper) Stop(rootDir string)

Stop stops the given scrape in a controlled manner This method will block until the scrape is stopped

func (*Scraper) StopAll

func (s *Scraper) StopAll()

StopAll stops all running scrapes and lets them exit in a controlled manner This method will block until all scrapes have stopped

type ScrapingFunc

type ScrapingFunc func(filename string, vid *models.Video, logger *logrus.Entry) error

A ScrapingFunc is a function that scrapes a file identified by its file name and writes the found meta data into the video struct provided

func MakeFileNameScraper

func MakeFileNameScraper(presetName string) (ScrapingFunc, error)

MakeFileNameScraper returns a scraping function that uses a regular expression to extract data from a file's name using capturing groups. These extracted data fields are then mapped to fields of the video struct, resulting in filling them with the appropriate data

Regex and field mappings are derived by taking them from the presets stored in this

func MustMakeFileNameScraper

func MustMakeFileNameScraper(presetName string) ScrapingFunc

MustMakeFileNameScraper is a version of MakeFileNameScraper that panics when creating the scraping function fails