Help us shape the future of Go! Take the 2026 Go Developer Survey

got

package module
v0.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 3, 2026 License: MIT Imports: 19 Imported by: 1

README

got

An HTTP client that downloads a file in concurrent byte ranges.

A fork of melbahja/got, extracted from skywire, where it lived as pkg/got and backs the skywire-cli got command. Skywire now imports it from here.

g := got.New()
g.ProgressFunc = func(d *got.Download) {
	fmt.Printf("\r%d / %d bytes  %s/s", d.Size(), d.TotalSize(), humanize(d.Speed()))
}
if err := g.Download("https://example.com/big.iso", "/tmp/big.iso"); err != nil {
	log.Fatal(err)
}

How a download runs

The first request is a one-byte range probe, Range: bytes=0-0, and what comes back decides everything after it.

If the server answers with a Content-Range, the file is rangeable. The size is read out of that header, the file is split into chunks, and the chunks are fetched concurrently into one output file that has already been truncated to its final length — each chunk writes at its own offset through an OffsetWriter, so there are no part files to concatenate afterwards and no second pass over the data.

If the server ignores the range and sends the whole body, the probe response is the download. It streams to disk right there rather than being discarded and refetched, so a server without range support costs one request, not two.

Concurrency and chunk size are derived from the file size and NumCPU when left at zero, and either can be pinned. Each chunk retries up to MaxRetries (3 by default) on its own, so one failed range does not fail the download.

Resuming

With Resume set, chunk completion is journaled to a state file beside the output. On a later run the journal is reloaded — after checking the URL still matches — and only the chunks that never finished are fetched again. The journal is removed once the download completes.

Resume is off by default, because it is only sound when the remote file has not changed underneath it.

SOCKS5

NewWithProxy takes a proxy address in three forms, and draws the same distinction curl does between the two schemes:

form who resolves the destination hostname
socks5h://host:port the proxy (curl --socks5-hostname)
socks5://host:port this client, before dialing (curl --socks5)
host:port the proxy — bare form is treated as socks5h://

The distinction is not cosmetic. A proxy that resolves names no DNS server knows — skywire's dmsgweb serves <pk>.dmsg hostnames this way — only works over socks5h, because that is the mode in which the hostname is handed to the proxy rather than looked up first. Under socks5://, an IP literal destination is passed straight through and only real hostnames take the resolver detour.

The rest of the API

A downloader needs an HTTP request layer anyway, so the plain one is exported rather than hidden:

  • Got.Request — any method, with headers and a body, response written to an io.Writer.
  • NewRequest, ParseHeaders ("Key: Value" strings), NormalizeURL (defaults a missing scheme to https), GetFilename.
  • Download fields and getters for progress reporting: Size, TotalSize, Speed, AvgSpeed, TotalCost.

A filename from a Content-Disposition header is rejected if it contains .., / or \, so a hostile server cannot steer the write outside the output directory.

Notes on the fork

Beyond the SOCKS5 support and the request helpers, the parts most changed from the original are about what happens concurrently:

  • Chunk-completion writes and the journal marshal are under one mutex; the progress-stop flag and the byte counters are atomic. The chunk goroutines and the progress goroutine both touch that state.
  • Got.Do propagates the receiver's Client and context into a Download built by hand. Without it a hand-built download silently fell back to DefaultClient(), which is how a configured SOCKS5 proxy could end up bypassed — the destination resolved locally instead of through the dialer NewWithProxy had wired up.
  • Do stops the progress goroutine and waits for it before returning, so no ProgressFunc call outlives the call that started it.

License

MIT — see LICENSE. Copyright (c) 2019 Mohamed Elbahja for the original, (c) 2026 Moses Narrow for this fork.

Dependency Graph

Made with goda:

go run github.com/loov/goda@latest graph github.com/0magnet/got/... | dot -Tsvg -o docs/got-goda-graph.svg

Dependency Graph

Lines of Code

Made with gocloc (excludes vendor/, node_modules/, .git/):

gocloc --not-match-d='(vendor|node_modules|\.git)' .

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                               8            217            174           1067
Markdown                         1             31              0            104
YAML                             1              0              7             98
Makefile                         1             19             34             85
Bourne Shell                     1              8             16             30
-------------------------------------------------------------------------------
TOTAL                           12            275            231           1384
-------------------------------------------------------------------------------

Documentation

Overview

Package got is an HTTP client that downloads a file in concurrent byte ranges.

A download starts with a one-byte range probe. If the server answers 206 with a Content-Range, the file is split into chunks and fetched in parallel into a single sparse output file, each chunk writing at its own offset through an OffsetWriter. If the server ignores the range — no Content-Range, or a 200 with the whole body — the probe response *is* the download, and it streams straight to disk rather than being thrown away and refetched.

Chunk progress is journaled beside the output file, so an interrupted download resumes from the chunks that finished rather than from zero.

The package also carries the plain request helpers a downloader needs anyway — Got.Request, NewRequest, ParseHeaders, NormalizeURL — and a SOCKS5 constructor, NewWithProxy, that distinguishes socks5:// from socks5h:// the way curl does. That distinction matters for proxies which resolve names no DNS server knows.

It began as a fork of github.com/melbahja/got and has since been substantially rewritten; see the README for what changed.

Index

Constants

This section is empty.

Variables

View Source 
var DefaultFileName = "got.output"

DefaultFileName is the fallback name when a filename cannot be derived from the URL.

View Source 
var ErrDownloadAborted = errors.New("download aborted")

ErrDownloadAborted is returned when a download is canceled.

View Source 
var UserAgent = "Got/2.0"

UserAgent is the default user agent for got HTTP requests.

Functions

func DefaultClient 

func DefaultClient() *http.Client

DefaultClient returns a new default HTTP client.

func GetFilename 

func GetFilename(URL string) string

GetFilename returns a filename derived from a URL path, or DefaultFileName.

func NewRequest 

func NewRequest(ctx context.Context, method, URL string, headers []Header, body io.Reader) (*http.Request, error)

NewRequest creates an http.Request with the given method, URL, headers, and optional body.

func NormalizeURL 

func NormalizeURL(rawURL string) (string, error)

NormalizeURL ensures a URL has a scheme, defaulting to https.

Types

type Chunk 

type Chunk struct {
	Start, End uint64
	Done       bool
}

Chunk represents a byte range for partial content download. Chunk and OffsetWriter are the byte-range primitives inherited from melbahja/got.

type Download 

type Download struct {
	Client *http.Client

	// Concurrency is the number of concurrent chunk downloads. 0 = auto.
	Concurrency uint

	// URL is the download URL.
	URL string

	// Dir is the output directory.
	Dir string

	// Dest is the output filename. If empty, derived from URL or Content-Disposition.
	Dest string

	// Interval controls progress callback frequency in milliseconds. 0 = auto.
	Interval uint64

	// ChunkSize in bytes. 0 = auto.
	ChunkSize uint64

	// MinChunkSize sets the minimum chunk size in bytes.
	MinChunkSize uint64

	// MaxChunkSize sets the maximum chunk size in bytes.
	MaxChunkSize uint64

	// MaxRetries per chunk. 0 = 3 (default).
	MaxRetries int

	// Resume enables resuming a previously interrupted download using state file.
	Resume bool

	// Header contains additional HTTP headers for the request.
	Header []Header

	// StopProgress signals the progress goroutine to stop. It is read by the
	// progress goroutine while another goroutine sets it, so access is atomic.
	StopProgress atomic.Bool
	// contains filtered or unexported fields
}

Download holds the configuration and state for a file download.

func NewDownload 

func NewDownload(ctx context.Context, URL, dest string) *Download

NewDownload creates a new Download with the given context, URL, and destination.

func (*Download) AvgSpeed 

func (d *Download) AvgSpeed() uint64

AvgSpeed returns the average download speed in bytes per second.

func (*Download) Context 

func (d *Download) Context() context.Context

Context returns the download context.

func (*Download) DownloadChunk 

func (d *Download) DownloadChunk(c *Chunk, dest io.Writer) error

DownloadChunk downloads a single byte range.

func (*Download) GetInfoOrDownload 

func (d *Download) GetInfoOrDownload() (*Info, error)

GetInfoOrDownload probes the server with a range request. If the server supports ranges (RFC 7233), it returns size info. Otherwise, it downloads the entire file in a single stream.

func (*Download) Init 

func (d *Download) Init() error

Init sets defaults, probes the server, and splits the file into chunks.

func (*Download) IsRangeable 

func (d *Download) IsRangeable() bool

IsRangeable returns whether the server supports partial content.

func (*Download) Path 

func (d *Download) Path() string

Path returns the output file path.

func (*Download) RunProgress 

func (d *Download) RunProgress(fn ProgressFunc)

RunProgress calls fn periodically until StopProgress is set.

func (*Download) Size 

func (d *Download) Size() uint64

Size returns the number of bytes downloaded so far.

func (*Download) Speed 

func (d *Download) Speed() uint64

Speed returns the current download speed in bytes per second.

func (*Download) Start 

func (d *Download) Start() error

Start downloads all chunks concurrently and writes them to the output file.

func (*Download) TotalCost 

func (d *Download) TotalCost() time.Duration

TotalCost returns how long the download has been running.

func (*Download) TotalSize 

func (d *Download) TotalSize() uint64

TotalSize returns the total file size (0 if unknown).

func (*Download) Write 

func (d *Download) Write(b []byte) (int, error)

Write implements io.Writer to track download progress.

type Got 

type Got struct {
	ProgressFunc

	// Client is the HTTP client used for requests. If nil, a default client is used.
	Client *http.Client
	// contains filtered or unexported fields
}

Got holds the configuration for HTTP operations.

func New 

func New() *Got

New returns a new Got with default context and client.

func NewWithContext 

func NewWithContext(ctx context.Context) *Got

NewWithContext returns a Got with the given context and default HTTP client.

func NewWithProxy 

func NewWithProxy(ctx context.Context, proxyAddr string) (*Got, error)

NewWithProxy returns a Got configured to route through a SOCKS5 proxy.

The proxyAddr is accepted in any of: 

socks5h://host:port  destination resolved by the proxy (curl --socks5-hostname).
                     Required for proxies that resolve non-DNS hostnames
                     themselves — e.g. dmsgweb's <pk>.dmsg URLs.
socks5://host:port   destination resolved by this client first, then the
                     IP is sent to the proxy (curl --socks5).
host:port            bare form, treated as socks5h:// for backward compat.

func (*Got) Do 

func (g *Got) Do(dl *Download) error

Do initializes and runs a Download, calling ProgressFunc if set.

Callers who construct a *Download by hand (e.g. `skywire-cli got dl`) typically don't set Client or ctx — without this propagation step, Download.Init falls back to DefaultClient(), silently discarding the proxy / context configured on the Got receiver. That manifests as the SOCKS5 proxy being bypassed (DNS lookups resolved locally instead of through the dialer wired up in NewWithProxy) — exactly the failure mode that turned out to outlive #3029.

func (*Got) Download 

func (g *Got) Download(URL, dest string) error

Download creates a Download for the given URL and destination, then runs it.

func (*Got) Request 

func (g *Got) Request(method, URL string, headers []Header, body io.Reader, output io.Writer) (*http.Response, error)

Request performs a general HTTP request and writes the response body to output. It returns the response (body already consumed) and any error. 

type Header struct {
	Key   string
	Value string
}

Header represents an HTTP header key-value pair.

func ParseHeaders 

func ParseHeaders(raw []string) ([]Header, error)

ParseHeaders parses header strings in "Key: Value" format.

type Info 

type Info struct {
	Size      uint64
	Rangeable bool
}

Info holds downloadable file info.

type OffsetWriter 

type OffsetWriter struct {
	io.WriterAt
	Offset int64
}

OffsetWriter wraps an io.WriterAt to implement io.Writer at a given offset.

func (*OffsetWriter) Write 

func (dst *OffsetWriter) Write(b []byte) (n int, err error)

Write implements io.Writer, writing at the current offset and advancing it.

type ProgressFunc 

type ProgressFunc func(d *Download)

ProgressFunc is called periodically during downloads to report progress.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.