download

Documentation

Overview

Package download transfers resolved media streams to files, writers, or callers that want an io.ReadCloser.

The package works with a Source: a signed URL plus the headers and metadata needed to fetch it. It does not depend on YouTube extraction; callers that can resolve or refresh a stream provide that through RefreshFunc.

Downloader.ToFile stages data in a temp file and commits it atomically. Downloader.ToWriter streams bytes directly to a caller writer. Downloader.Stream returns the response body behind an io.ReadCloser.

Signed stream URLs can expire during transfer. On HTTP 403 or 410, the downloader asks the refresh function for a new Source and retries the affected range. Without a refresh function, expiry fails with waxerr.ErrURLExpired.

Index

• type Config
• type Downloader
  • func New(cfg Config) *Downloader
  • func (d *Downloader) Stream(ctx context.Context, src Source, refresh RefreshFunc, progress ProgressFunc) (io.ReadCloser, StreamInfo, error)
  • func (d *Downloader) ToFile(ctx context.Context, src Source, path string, refresh RefreshFunc, ...) (Result, error)
  • func (d *Downloader) ToWriter(ctx context.Context, src Source, w io.Writer, refresh RefreshFunc, ...) (Result, error)
• type HeaderRange
  • func (HeaderRange) Apply(req *http.Request, start, end int64)
  • func (HeaderRange) Validate(resp *http.Response, start, end int64) error
• type Progress
• type ProgressFunc
• type QueryRange
  • func (QueryRange) Apply(req *http.Request, start, end int64)
  • func (QueryRange) Validate(resp *http.Response, start, end int64) error
• type RangeStrategy
• type RefreshFunc
• type Result
• type Source
• type StreamInfo

Types

type Config

type Config struct {
	// HTTPClient performs requests with retry/backoff and rate-limit handling.
	// It is required.
	HTTPClient *httpx.Client
	// Logger receives debug logs. Nil discards them.
	Logger *slog.Logger

	// ChunkSize is the byte length of a single ranged chunk (default 10 MiB).
	ChunkSize int64
	// Parallelism caps simultaneous chunks within one ToFile download (default 4).
	Parallelism int
	// ChunkTimeout bounds one ToFile chunk request, including headers and body. 0
	// means no extra deadline beyond the request context. Streaming methods do
	// not apply this timeout.
	ChunkTimeout time.Duration

	// MaxChunkRetries is the per-chunk retry budget for transient failures after
	// the first request (default 3). Expiry refreshes are counted separately.
	MaxChunkRetries int
	// MaxRefreshes caps URL refreshes per download (default 2).
	MaxRefreshes int

	// BaseBackoff and MaxBackoff tune the download-layer retry sleep (distinct
	// from httpx's own HTTP-level backoff).
	BaseBackoff time.Duration
	MaxBackoff  time.Duration
}

Config configures a Downloader. The zero value is usable except for HTTPClient, which is required; New fills the rest with defaults.

type Downloader

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader fetches Sources into sinks. It is safe for concurrent use.

func New

func New(cfg Config) *Downloader

New returns a Downloader, filling unset Config fields with defaults. It panics if HTTPClient is nil, since a Downloader cannot function without one.

func (*Downloader) Stream

func (d *Downloader) Stream(ctx context.Context, src Source, refresh RefreshFunc, progress ProgressFunc) (io.ReadCloser, StreamInfo, error)

Stream returns an io.ReadCloser for src. The reader resumes from its current offset after an expired URL or a transient mid-stream failure. The first request is issued before Stream returns so early failures surface here instead of on the first Read.

The caller must keep ctx alive until Close and must Close the reader; final byte counts are known only after the reader is drained. refresh and progress may be nil.

func (*Downloader) ToFile

func (d *Downloader) ToFile(ctx context.Context, src Source, path string, refresh RefreshFunc, progress ProgressFunc) (Result, error)

ToFile downloads src to path. When the content length is known and larger than the configured chunk size, it writes byte ranges in parallel through io.WriterAt; otherwise it uses a single resumable stream.

Data is staged in a temp file next to path and renamed into place only after the download completes. Failed or canceled downloads remove the temp file and do not leave path behind.

refresh re-resolves the URL if it expires mid-download; progress receives best-effort byte updates. Both may be nil.

func (*Downloader) ToWriter

func (d *Downloader) ToWriter(ctx context.Context, src Source, w io.Writer, refresh RefreshFunc, progress ProgressFunc) (Result, error)

ToWriter streams src to w in order, with bounded memory and no temp file. If the URL expires, the downloader refreshes the Source and resumes from the number of bytes already written.

Unlike ToFile, ToWriter gives no atomicity guarantee: bytes already handed to w are not retractable if a later error occurs. refresh and progress may be nil.

type HeaderRange

type HeaderRange struct{}

HeaderRange requests ranges with the standard Range header and expects a 206 Partial Content response. It is the default strategy.

func (HeaderRange) Apply

func (HeaderRange) Apply(req *http.Request, start, end int64)

Apply sets "Range: bytes=start-end" (open-ended when end < 0).

func (HeaderRange) Validate

func (HeaderRange) Validate(resp *http.Response, start, end int64) error

Validate accepts 206 for any range. A 200 is safe only for bytes=0-, where the full body starts at the requested offset. For bounded chunks and resumes, 200 means the server ignored the Range header and would corrupt an offset write.

type Progress

type Progress struct {
	BytesWritten int64 // bytes delivered to the sink so far
	Total        int64 // total expected bytes, or 0 if unknown
}

Progress is a byte-count snapshot delivered to a ProgressFunc.

type ProgressFunc

type ProgressFunc func(Progress)

ProgressFunc receives best-effort byte progress. It is called synchronously from a download worker, so it must be fast and must not block. It may be nil.

type QueryRange

type QueryRange struct{}

QueryRange requests ranges with a &range=start-end query parameter and expects a 200 response carrying the requested bytes.

func (QueryRange) Apply

func (QueryRange) Apply(req *http.Request, start, end int64)

Apply sets the range query parameter (open-ended when end < 0).

func (QueryRange) Validate

func (QueryRange) Validate(resp *http.Response, start, end int64) error

Validate accepts 200 and verifies the declared range or length when the response provides enough information. Open-ended requests have no expected byte count, so Content-Length alone is not useful.

type RangeStrategy

type RangeStrategy interface {
	// Apply adds the requested range to req. The downloader calls it for every
	// fetch, including the initial open-ended bytes=0- request.
	Apply(req *http.Request, start, end int64)
	// Validate rejects responses that cannot be safely copied to the requested
	// offset: wrong status, ignored range, or conflicting range/length headers.
	Validate(resp *http.Response, start, end int64) error
}

RangeStrategy describes the wire format for byte ranges and the checks needed before the downloader writes bytes at an offset. Standard HTTP servers use a Range header and answer 206; some media origins expect a &range= query parameter and answer 200 with the requested bytes.

Ranges are inclusive: [start, end] selects end-start+1 bytes. An end < 0 means "to the end of the resource" (open-ended), used by the streaming sinks for the initial bytes=0- GET and when resuming from an offset.

type RefreshFunc

type RefreshFunc func(ctx context.Context, failure *potoken.HTTPFailure) (Source, error)

RefreshFunc returns a replacement Source after a request indicates that the current source expired. failure contains the HTTP status and a bounded body sample from the response that triggered the refresh. A nil RefreshFunc disables refresh; expired URLs then fail with waxerr.ErrURLExpired.

type Result

type Result struct {
	BytesWritten int64
}

Result reports the byte count for a completed download.

type Source

type Source struct {
	// URL is the signed, playable stream URL.
	URL string
	// ContentLength is the total size in bytes, or 0 if unknown. When known and
	// large enough, ToFile downloads it in parallel ranges; when unknown, all
	// sinks fall back to a single streamed GET.
	ContentLength int64
	// Headers are sent on every request. Some origins bind a signed URL to the
	// request identity used during resolution.
	Headers http.Header
	// ExpiresAt is when the signed URL is expected to lapse, if known. It is
	// advisory: the authoritative signal is a 403 on the wire, which triggers a
	// refresh regardless of this value.
	ExpiresAt time.Time
	// RangeStrategy controls how byte ranges are requested and validated. Nil
	// selects HeaderRange. Use QueryRange for origins that expect a &range= query
	// parameter instead of a Range header.
	RangeStrategy RangeStrategy
}

Source is the input to a download: a resolved, signed URL plus the metadata needed to request it.

type StreamInfo

type StreamInfo struct {
	ContentLength int64  // total size in bytes, or 0 if unknown
	ContentType   string // Content-Type of the first response, if any
}

StreamInfo is the response metadata returned by Stream alongside the reader.