grabber

README

Grabber

A Go library for downloading media from various platforms using yt-dlp

Table of Contents

1. About
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About

Grabber is a Go library that provides a simple and unified interface for downloading media from various social media platforms including TikTok, Instagram, Twitter/X, and YouTube. Built on top of the powerful yt-dlp tool, it offers a clean Go API with configurable options for download behavior, platform detection, and support for both photos and videos.

Key features include:

• Generic interface for downloading media from supported platforms
• Configurable options for download behavior
• Automatic platform detection
• Support for photos, videos, and GIFs
• Built on top of github.com/lrstanley/go-ytdlp

(back to top)

Built With

• Go
• yt-dlp
• go-ytdlp

(back to top)

Getting Started

To get started with Grabber, you'll need to have Go installed on your system and ensure that the required dependencies are available.

Prerequisites

• Go 1.24.3 or later
• yt-dlp (automatically handled by the library)
• Optional: ffmpeg (required for video processing)

Installation

Install the library using Go modules:

go get github.com/ethanbaker/grabber

Other prerequisites, such as ffmpeg, can be installed for specific post processing/reencoding tasks. For example, ffmpeg is used to reencode mp4 videos being sent to iOS from telegram. To install ffmpeg for linux, run:

sudo apt install ffmpeg

(back to top)

Usage

Basic Usage

Here's a simple example of how to use Grabber to download media:

package main

import (
    "fmt"
    "log"
    "os"
    "time"

    "github.com/ethanbaker/grabber"
)

func main() {
    // Create a grabber with default options
    g := grabber.New()

    // Check if URL was provided as argument
    if len(os.Args) < 2 {
        log.Fatal("Usage: go run main.go <url>")
    }

    url := os.Args[1]

    // Identify the platform
    platform := g.IdentifyPlatform(url)
    if platform == grabber.Unknown {
        log.Fatal("Unsupported platform for URL:", url)
    }

    fmt.Printf("Detected platform: %s\n", platform)
    fmt.Printf("Downloading from: %s\n", url)

    // Download the media
    name := fmt.Sprintf("download-%s", time.Now().Format("20060102_150405"))
    result, err := g.Download(url, name)
    if err != nil {
        log.Fatalf("Download failed: %v", err)
    }

    // Print the results
    for _, file := range result {
        fmt.Printf("Successfully downloaded file %s!\n", file.Name)
        fmt.Printf("\tPath: %s\n", file.Path)
        fmt.Printf("\tType: %s\n", file.MediaType)
    }

    // Perform post download operations based on platform
    // ...
}

Custom Configuration

You can customize the download behavior by providing custom options:

package main

import (
    "github.com/ethanbaker/grabber"
)

func main() {
    // Create grabber with a few tweaked options
    g := grabber.New().WithBaseDir("downloads/").
            WithMaxFileSize("50M").
            WithNoPlaylist(true)

    // ----------------------------------------

    // Create a grabber with fully custom options
    options := &grabber.Options{
        BaseDir:           "./my-downloads/",
        MaxFileSize:       "100M",
        Format:            "best",
        RestrictFilenames: true,
        NoPlaylist:        true,
        WriteThumbnail:    false,
        OutputTemplate:    "%(title)s.%(ext)s",
    }
    
    g := grabber.New().WithOptions(options)
}

The default options for a grabber are:

&grabber.Options{
    BaseDir:           "media/",
    MaxFileSize:       "49M",
    Format:            "bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best",
    RestrictFilenames: true,
    NoPlaylist:        true,
    WriteThumbnail:    true,
    RecodeVideo:       "",
    OutputTemplate:    "media.%(ext)s",
}

Telegram Bot Integration

The library includes an example Telegram bot implementation that demonstrates how to integrate Grabber into a messaging application. The bot can receive URLs from users and automatically download and send the media back. This is particularly useful for creating bots that can archive or redistribute content from various social media platforms.

To run the Telegram bot example:

1. Create a bot token through @BotFather
2. Set up your environment with the bot token
3. Run the bot and send it URLs from supported platforms

Supported Platforms

• TikTok
• Instagram Reels
• Twitter/X
• YouTube

For more examples, please refer to the documentation.

(back to top)

Roadmap

• Core downloading functionality
• Platform detection
• Configurable options
• Telegram bot example
• Additional platform support
• Discord implementation

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

For issues and suggestions, please include as much useful information as possible. Review the documentation and make sure the issue is actually present or the suggestion is not included. Please share issues/suggestions on the issue tracker.

For patches and feature additions, please submit them as pull requests. Please adhere to the conventional commits standard for commit messaging. In addition, please try to name your git branch according to your new patch. These standards are a great guide you can follow.

You can follow these steps below to create a pull request:

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/amazing-feature)
3. Commit your Changes (git commit -m "feat: add amazing feature")
4. Push to the Branch (git push origin feature/amazing-feature)
5. Open a Pull Request

(back to top)

License

This project is licensed under the MIT License - see the LICENSE file for details.

(back to top)

Contact

Ethan Baker - contact@ethanbaker.dev - LinkedIn

Project Link: https://github.com/ethanbaker/grabber

(back to top)

Acknowledgments

• yt-dlp - The powerful media downloader that powers this library
• go-ytdlp - Go bindings for yt-dlp

(back to top)

Documentation

Overview

utility functions

Index

• func ReencodeForTelegramIOS(path string) error
• type FileResult
• type Grabber
  • func New() *Grabber
  • func (g *Grabber) Download(mediaUrl string, name string) ([]FileResult, error)
  • func (g *Grabber) IdentifyPlatform(webUrl string) Platform
  • func (g *Grabber) WithBaseDir(baseDir string) *Grabber
  • func (g *Grabber) WithFormat(format string) *Grabber
  • func (g *Grabber) WithMaxFileSize(size string) *Grabber
  • func (g *Grabber) WithNoPlaylist(noPlaylist bool) *Grabber
  • func (g *Grabber) WithOptions(options *Options) *Grabber
  • func (g *Grabber) WithOutputTemplate(template string) *Grabber
  • func (g *Grabber) WithRecodeVideo(format string) *Grabber
  • func (g *Grabber) WithRestrictFilenames(restrict bool) *Grabber
  • func (g *Grabber) WithWriteThumbnail(write bool) *Grabber
• type MediaType
• type Options
• type Platform

Functions

func ReencodeForTelegramIOS

func ReencodeForTelegramIOS(path string) error

ReencodeForIOS takes input and output file paths and re-encodes the video to be iOS/Telegram-compatible.

Types

type FileResult

type FileResult struct {
	Name      string
	Path      string
	Ext       string
	MediaType MediaType
}

FileResult represents the result of a download operation

type Grabber

type Grabber struct {
	// contains filtered or unexported fields
}

Grabber handles media downloading functionality

func New

func New() *Grabber

New creates a new Grabber instance

func (*Grabber) Download

func (g *Grabber) Download(mediaUrl string, name string) ([]FileResult, error)

Download downloads media from the provided URL using yt-dlp

func (*Grabber) IdentifyPlatform

func (g *Grabber) IdentifyPlatform(webUrl string) Platform

IdentifyPlatform checks the domain of the URL to determine the platform

func (*Grabber) WithBaseDir

func (g *Grabber) WithBaseDir(baseDir string) *Grabber

Only set base directory for the grabber

func (*Grabber) WithFormat

func (g *Grabber) WithFormat(format string) *Grabber

WithFormat sets the format preference for yt-dlp

func (*Grabber) WithMaxFileSize

func (g *Grabber) WithMaxFileSize(size string) *Grabber

WithMaxFileSize sets the maximum file size to download

func (*Grabber) WithNoPlaylist

func (g *Grabber) WithNoPlaylist(noPlaylist bool) *Grabber

WithNoPlaylist sets whether to download only a single video/image if the URL is part of a playlist

func (*Grabber) WithOptions

func (g *Grabber) WithOptions(options *Options) *Grabber

WithOptions creates a new Grabber instance with the provided options

func (*Grabber) WithOutputTemplate

func (g *Grabber) WithOutputTemplate(template string) *Grabber

WithOutputTemplate sets the output filename template

func (*Grabber) WithRecodeVideo

func (g *Grabber) WithRecodeVideo(format string) *Grabber

WithRecodeVideo sets the video recoding format

func (*Grabber) WithRestrictFilenames

func (g *Grabber) WithRestrictFilenames(restrict bool) *Grabber

WithRestrictFilenames enables or disables filename sanitization

func (*Grabber) WithWriteThumbnail

func (g *Grabber) WithWriteThumbnail(write bool) *Grabber

WithWriteThumbnail enables or disables thumbnail downloading

type MediaType

type MediaType string

MediaType represents the type of media downloaded

const (
	Photo     MediaType = "photo"
	Video     MediaType = "video"
	Animation MediaType = "animation"
	None      MediaType = "none"
)

type Options

type Options struct {
	BaseDir           string // BaseDir is the directory where downloaded files will be saved
	MaxFileSize       string // MaxFileSize is the maximum file size to download (e.g., "49M" for 49 megabytes)
	Format            string // Format specifies the format preference for yt-dlp
	RestrictFilenames bool   // RestrictFilenames sanitizes filenames if true
	RecodeVideo       string // RecodeVideo specifies the video format to recode to (e.g., "mp4")
	NoPlaylist        bool   // NoPlaylist downloads only single video/image if URL is part of a playlist
	WriteThumbnail    bool   // WriteThumbnail attempts to download thumbnail if true
	OutputTemplate    string // OutputTemplate specifies the output filename template
}

Options represents configuration options for the Grabber

type Platform

type Platform string

Platform represents the supported platforms grabber can download

const (
	Tiktok         Platform = "tiktok"
	InstagramReels Platform = "instagram reels"
	Twitter        Platform = "twitter"
	YouTube        Platform = "youtube"
	Unknown        Platform = "unknown"
)