rtve

README

RTVE Scraper

A command-line tool for downloading subtitles and video metadata from RTVE (Radio Televisión Española).

Features

• Scrape videos from RTVE show pages
• Download video metadata in JSON format
• Download subtitles in VTT format
• Organize videos by publication date
• Support for rate limiting and pagination
• Command-line interface with customizable options

Installation

Prerequisites

• Go 1.23 or higher

Building from source

go install github.com/rubiojr/rtve-go/cmd/rtve-subs@latest

Usage

rtve-subs fetch --show telediario-1

# Specify output directory
rtve-subs --output="/path/to/videos" --show telediario-1

# Enable verbose output
rtve-subs --verbose --show telediario-1

Supported Shows

Currently supported shows include:

• telediario-1
• telediario-2
• informe-semanal

Use the list-shows command to see the complete list of available shows

Command-line Options

Option	Alias	Default	Description
--output	-o	rtve-videos	Output directory for downloaded content
--show	-p	none	Show to scrap (required)
--max-pages	-m	1024	Maximum number of pages to scrape
--verbose	-v	false	Enable verbose output

Output Structure

The scraper organizes videos by year and date:

rtve-videos/
  ├── 2023/
  │   ├── 2023-01-01/
  │   │   ├── video_12345.json
  │   │   └── subs/
  │   │       ├── 12345_es.vtt
  │   │       └── 12345_en.vtt
  │   └── 2023-01-02/
  │       └── ...
  └── 2022/
      └── ...

How It Works

1. The scraper navigates through pages of the specified RTVE show URL
2. For each page, it extracts links to individual videos
3. For each video:
   • Metadata is downloaded and saved as JSON
   • Subtitles are downloaded if available
   • Content is organized by publication date

License

This project is licensed under the MIT License - see the LICENSE file for details.

Disclaimer

This tool is provided for personal use only. Be mindful of RTVE's terms of service and copyright restrictions when using this tool. The developers are not responsible for any misuse of this software.

Documentation

Index

• Constants
• Variables
• func GetLanguageName(langCode string) string
• func ListShows() []string
• type Option
  • func WithOutputPath(path string) Option
• type Scrapper
  • func NewScrapper(program string, options ...Option) *Scrapper
  • func (s *Scrapper) DownloadSubtitles(meta *VideoMetadata, outputDir string) error
  • func (s *Scrapper) DownloadVideoMeta(videoID string) (*VideoMetadata, error)
  • func (s *Scrapper) FetchSubtitles(id string) (*SubtitleResponse, error)
  • func (s *Scrapper) SaveVideoToFile(meta *VideoMetadata, directory string) error
  • func (s *Scrapper) Scrape(maxPages int) (int, []error)
  • func (s *Scrapper) ScrapePage(page int) ([]*VideoInfo, error)
• type Show
  • func ShowMap(name string) *Show
• type SubtitleItem
• type SubtitlePage
• type SubtitleResponse
• type VideoInfo
• type VideoMetadata
  • func (m *VideoMetadata) Parse(body string) error
• type VideoPage
• type VideoResponse

Constants

const ApiURL = "https://api2.rtve.es/api/videos/%s.json"

const SubsURL = "https://api2.rtve.es/api/videos/%s/subtitulos.json"

Variables

var ErrForbidden = errors.New("access not allowed")

var ErrPageNotFound = errors.New("page not found")

Functions

func GetLanguageName

func GetLanguageName(langCode string) string

Helper function to get language name from language code

func ListShows

func ListShows() []string

Types

type Option

type Option func(*Scrapper)

func WithOutputPath

func WithOutputPath(path string) Option

type Scrapper

type Scrapper struct {
	Program string
	// contains filtered or unexported fields
}

func NewScrapper

func NewScrapper(program string, options ...Option) *Scrapper

func (*Scrapper) DownloadSubtitles

func (s *Scrapper) DownloadSubtitles(meta *VideoMetadata, outputDir string) error

DownloadSubtitles downloads all available subtitles for a given video ID and saves them to the specified directory

func (*Scrapper) DownloadVideoMeta

func (s *Scrapper) DownloadVideoMeta(videoID string) (*VideoMetadata, error)

DownloadVideoMeta fetches and parses video metadata for a given video ID

func (*Scrapper) FetchSubtitles

func (s *Scrapper) FetchSubtitles(id string) (*SubtitleResponse, error)

func (*Scrapper) SaveVideoToFile

func (s *Scrapper) SaveVideoToFile(meta *VideoMetadata, directory string) error

func (*Scrapper) Scrape

func (s *Scrapper) Scrape(maxPages int) (int, []error)

func (*Scrapper) ScrapePage

func (s *Scrapper) ScrapePage(page int) ([]*VideoInfo, error)

type Show

type Show struct {
	ID    string
	URL   string
	Regex string
}

func ShowMap

func ShowMap(name string) *Show

type SubtitleItem

type SubtitleItem struct {
	Src  string `json:"src"`  // URL of the subtitle file
	Lang string `json:"lang"` // Language code for the subtitle
}

type SubtitlePage

type SubtitlePage struct {
	Items       []SubtitleItem `json:"items"`       // List of subtitle items
	Number      int            `json:"number"`      // Current page number
	Size        int            `json:"size"`        // Page size
	Offset      int            `json:"offset"`      // Offset in the results
	Total       int            `json:"total"`       // Total number of items
	TotalPages  int            `json:"totalPages"`  // Total number of pages
	NumElements int            `json:"numElements"` // Number of elements in the current page
}

SubtitlePage represents the page of subtitle items along with pagination info

type SubtitleResponse

type SubtitleResponse struct {
	Page SubtitlePage `json:"page"` // Subtitle page information
}

SubtitleResponse represents the top-level JSON response

type VideoInfo

type VideoInfo struct {
	URL string
	ID  string
}

type VideoMetadata

type VideoMetadata struct {
	URI             string `json:"uri"`
	HTMLUrl         string `json:"htmlUrl"`
	ID              string `json:"id"`
	LongTitle       string `json:"longTitle"`
	PublicationDate string `json:"publicationDate"`
}

VideoMetadata represents essential metadata from a video

func (*VideoMetadata) Parse

func (m *VideoMetadata) Parse(body string) error

type VideoPage

type VideoPage struct {
	Items       []VideoMetadata `json:"items"`
	Number      int             `json:"number"`
	Size        int             `json:"size"`
	Offset      int             `json:"offset"`
	Total       int             `json:"total"`
	TotalPages  int             `json:"totalPages"`
	NumElements int             `json:"numElements"`
}

VideoPage represents the page of video items

type VideoResponse

type VideoResponse struct {
	Page VideoPage `json:"page"`
}

VideoResponse represents the top-level JSON response