dlfetch

README

dlfetch

dlfetch is a Go library designed to make file downloading easy and efficient. It uses Go's concurrency to handle multiple downloads, managed by a number of configurable workers. It also includes error handling and on completion callbacks.

Usage Example

package main

import "github.com/hritikr/dlfetch"

func main() {
    // Create a new Fetcher instance with default configuration
    fetcher := dlfetch.New()
    
    // Add a download request to the Fetcher's queue
    fetcher.Enqueue(dlfetch.DownloadRequest{
        ID: "example_file",
        URL: "https://example.com/file",
    })

    // Start processing download requests
    fetcher.Start()

    // When finished, stop the Fetcher
    fetcher.Stop()
}

For a more complete usage example, see the example/example.go in this repository.

Functionality

dlfetch includes versatile configurability via functional options that allow you to:

• Change the default HTTP client
• Set the number of concurrent workers
• Specify the directory where downloaded files are saved
• Define custom behavior when a download completes or encounters an error

You can also add and manage multiple download requests at once using the EnqueueMany() function.

Installation

go get -u github.com/hritikr/dlfetch

Contribution

For any issues or to provide feedback, please open an issue on this repository. We also welcome contributions. If you'd like to make changes or improve this library, please feel free to make a pull request.

This project is licensed under the MIT License. See LICENSE for more details.

Documentation

Index

• Constants
• type DownloadRequest
• type DownloadResult
  • func (d *DownloadResult) IsAudio() bool
  • func (d *DownloadResult) IsImage() bool
  • func (d *DownloadResult) IsVideo() bool
• type DownloadStatus
• type DownloadTask
• type EnqueueResult
• type Fetcher
  • func New(options ...FetcherOption) *Fetcher
  • func (f *Fetcher) Enqueue(req DownloadRequest) EnqueueResult
  • func (f *Fetcher) EnqueueMany(reqs []DownloadRequest) []EnqueueResult
  • func (f *Fetcher) Start()
  • func (f *Fetcher) Stop()
• type FetcherOption
  • func WithEnableOverwrite(eo bool) FetcherOption
  • func WithHTTPClient(client *http.Client) FetcherOption
  • func WithMaxWorkers(max int) FetcherOption
  • func WithMonitor(m Monitor) FetcherOption
  • func WithOnComplete(callback func(DownloadResult)) FetcherOption
  • func WithOnError(callback func(DownloadRequest, error)) FetcherOption
  • func WithTargetDir(dir string) FetcherOption
• type Monitor
• type MonitorSnapshot
• type TaskMonitor
  • func NewMonitor() *TaskMonitor
  • func (m *TaskMonitor) EventSignal() <-chan struct{}
  • func (m *TaskMonitor) GetSnapshot() MonitorSnapshot
• type TaskStatusCount

Constants

const UnknownSize int64 = -1

Types

type DownloadRequest

type DownloadRequest struct {
	ID       int
	URL      string
	FileName string
	Path     string // Path will be optional; if empty, use only FileName and targetDir
	MimeType string
	FullPath string // Computed after enqueuing
}

type DownloadResult

type DownloadResult struct {
	ID       int
	FileName string
	Path     string
	MimeType string
}

func (*DownloadResult) IsAudio

func (d *DownloadResult) IsAudio() bool

func (*DownloadResult) IsImage

func (d *DownloadResult) IsImage() bool

func (*DownloadResult) IsVideo

func (d *DownloadResult) IsVideo() bool

type DownloadStatus

type DownloadStatus string

Download Monitoring

const (
	StatusPending    DownloadStatus = "pending"
	StatusInProgress DownloadStatus = "in_progress"
	StatusCompleted  DownloadStatus = "completed"
	StatusFailed     DownloadStatus = "failed"
)

type DownloadTask

type DownloadTask struct {
	ID            int            `json:"id"`
	FileName      string         `json:"fileName"`
	FilePath      string         `json:"filePath"`
	TotalBytes    int64          `json:"totalBytes"`
	DoneBytes     int64          `json:"doneBytes"`
	Status        DownloadStatus `json:"status"`
	Error         string         `json:"error,omitempty"`
	StartedAt     time.Time      `json:"startedAt"`
	CompletedAt   *time.Time     `json:"completedAt,omitempty"`
	DownloadSpeed float64        `json:"downloadSpeed"`
	ETA           string         `json:"eta"`
	QueuePosition int            `json:"queuePosition"`
	EnqueuedAt    time.Time      `json:"enqueuedAt"`
}

type EnqueueResult

type EnqueueResult struct {
	Request DownloadRequest
	Queued  bool
	Error   error
}

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher is responsible for managing download requests and processing them. It supports configuration through functional options.

func New

func New(options ...FetcherOption) *Fetcher

New creates a new Fetcher instance with the provided options.

func (*Fetcher) Enqueue

func (f *Fetcher) Enqueue(req DownloadRequest) EnqueueResult

Enqueue adds a download request to the Fetcher's queue.

func (*Fetcher) EnqueueMany

func (f *Fetcher) EnqueueMany(reqs []DownloadRequest) []EnqueueResult

EnqueueMany adds multiple download requests to the Fetcher's queue.

func (*Fetcher) Start

func (f *Fetcher) Start()

Start begins processing download requests with the configured number of workers.

func (*Fetcher) Stop

func (f *Fetcher) Stop()

Stop signals the Fetcher to stop processing and waits for all workers to finish. Closes the monitor's event signal

type FetcherOption

type FetcherOption func(*Fetcher)

FetcherOption defines a function type for configuring the Fetcher. Each option function modifies the Fetcher's fields.

func WithEnableOverwrite

func WithEnableOverwrite(eo bool) FetcherOption

WithEnableOverwrite sets the enableOverwrite for the Fetcher When enabled the files gets overwritten if they exists

func WithHTTPClient

func WithHTTPClient(client *http.Client) FetcherOption

WithHTTPClient sets a custom HTTP client for the Fetcher.

func WithMaxWorkers

func WithMaxWorkers(max int) FetcherOption

WithMaxWorkers sets the maximum number of concurrent workers for the Fetcher.

func WithMonitor

func WithMonitor(m Monitor) FetcherOption

WithMonitor sets the Monitor for the Fetcher.

func WithOnComplete

func WithOnComplete(callback func(DownloadResult)) FetcherOption

WithOnComplete sets the callback function to be called on download completion.

func WithOnError

func WithOnError(callback func(DownloadRequest, error)) FetcherOption

WithOnError sets the callback function to be called on download error.

func WithTargetDir

func WithTargetDir(dir string) FetcherOption

WithTargetDir sets the target directory for downloaded files.

type Monitor

type Monitor interface {
	GetSnapshot() MonitorSnapshot
	EventSignal() <-chan struct{}
	// contains filtered or unexported methods
}

type MonitorSnapshot

type MonitorSnapshot struct {
	Tasks []DownloadTask  `json:"tasks"`
	Count TaskStatusCount `json:"count"`
}

type TaskMonitor

type TaskMonitor struct {
	// contains filtered or unexported fields
}

func NewMonitor

func NewMonitor() *TaskMonitor

Creates a TaskMonitor

func (*TaskMonitor) EventSignal

func (m *TaskMonitor) EventSignal() <-chan struct{}

EventSignal returns a read-only channel that signals whenever the TaskMonitor's state changes.

func (*TaskMonitor) GetSnapshot

func (m *TaskMonitor) GetSnapshot() MonitorSnapshot

GetSnapshot returns a copy of the current state of all download

type TaskStatusCount

type TaskStatusCount struct {
	Total      int `json:"total"`
	Pending    int `json:"pending"`
	InProgress int `json:"in_progress"`
	Completed  int `json:"completed"`
	Failed     int `json:"failed"`
}