url

Documentation

Overview

Package url implements a hapiq downloader that fetches directly from an arbitrary HTTP(S) URL. The "ID" is the URL itself. In manifests use the dedicated url: field:

• identifier: my-file url: https://example.com/file.csv

A URL whose path ends in "/" is treated as a directory: its files are enumerated from the server's JSON index (Caddy file_server browse, nginx autoindex_format json) and fetched into the output directory, subject to the usual --include-ext / --filename-pattern / --limit-files filters.

• identifier: my-dataset url: https://example.com/datasets/be1-fixture/

Index

• type Option
  • func WithTimeout(t time.Duration) Option
  • func WithVerbose(v bool) Option
• type URLDownloader
  • func New(opts ...Option) *URLDownloader
  • func (d *URLDownloader) Download(ctx context.Context, req *downloaders.DownloadRequest) (*downloaders.DownloadResult, error)
  • func (d *URLDownloader) GetMetadata(ctx context.Context, id string) (*downloaders.Metadata, error)
  • func (d *URLDownloader) GetSourceType() string
  • func (d *URLDownloader) Validate(_ context.Context, id string) (*downloaders.ValidationResult, error)

Types

type Option

type Option func(*URLDownloader)

Option configures a URLDownloader.

func WithTimeout

func WithTimeout(t time.Duration) Option

WithTimeout sets the HTTP client timeout.

func WithVerbose

func WithVerbose(v bool) Option

WithVerbose toggles progress logging to stderr.

type URLDownloader

type URLDownloader struct {
	// contains filtered or unexported fields
}

URLDownloader downloads a single file from a user-supplied HTTP(S) URL.

func New

func New(opts ...Option) *URLDownloader

New creates a URLDownloader.

func (*URLDownloader) Download

func (d *URLDownloader) Download(ctx context.Context, req *downloaders.DownloadRequest) (*downloaders.DownloadResult, error)

Download fetches req.ID into req.OutputDir: one file, or -- when the URL addresses a directory -- every file its index lists.

func (*URLDownloader) GetMetadata

func (d *URLDownloader) GetMetadata(ctx context.Context, id string) (*downloaders.Metadata, error)

GetMetadata issues a HEAD request to retrieve size and content-type.

func (*URLDownloader) GetSourceType

func (d *URLDownloader) GetSourceType() string

GetSourceType returns the source identifier.

func (*URLDownloader) Validate

func (d *URLDownloader) Validate(_ context.Context, id string) (*downloaders.ValidationResult, error)

Validate checks that id is a valid http or https URL.