vision

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 27, 2026 License: BSD-3-Clause Imports: 18 Imported by: 0

Documentation

Overview

Package vision is the only package in this library that speaks HTTP.

It talks to three endpoints, which are three genuinely different services that happen to share a domain name: 

data.binance.vision                   fetches one archive        download.go
s3-ap-northeast-1.amazonaws.com/...   lists what exists          listing.go
data-api.binance.vision               the recent tail, as JSON   klines.go

The first two are a static file server and the S3 API in front of it, and neither has a quota. The third is the read-only half of Binance's trading API — market data, no key — and it enforces the trading API's rate limit, which is why limiter.go exists and applies to that endpoint alone.

Everything here is transport. No type in this package knows what a candle is: strings and bytes go in, strings and bytes come out. That is what allows it to live under internal/ at all — see docs/architecture.md, which draws the line at whether a package needs the root's domain types.

What the bucket is

data.binance.vision is not a Binance service. It is an ordinary Amazon S3 bucket, literally named "data.binance.vision", in the ap-northeast-1 region, which Binance pointed a domain at and left publicly readable. That detail is invisible when downloading a file and unavoidable when asking what files exist, because the two go through different front doors: 

https://data.binance.vision/data/spot/monthly/klines/...zip
    fetches one object. Binance's domain, Binance's problem to keep working.

https://s3-ap-northeast-1.amazonaws.com/data.binance.vision?prefix=...
    lists objects. S3's own API, and it bakes in the provider, the region
    and the bucket name.

Only the second one can answer "what exists?". Verified on 2026-08-18: requesting the listing query string from data.binance.vision returns the HTML file-browser page — a static page that calls this same S3 API from JavaScript — with HTTP 200 and content-type text/html. There is no way to list through Binance's domain.

Why that is acceptable anyway

Because listing is an optimisation and never the source of truth. If Binance migrates the bucket, this package starts returning errors and the layers above degrade to fetching what they were going to fetch anyway; what they must never do is read a failed listing as "no data exists". That is why Lister.List distinguishes three outcomes rather than two, and why the empty listing is the one that needs explaining — see Lister.List.

The base URL is a field rather than a constant so that a consumer can re-point it without waiting for a release, and so that every test in this package can aim it at an httptest.Server.

Index

Constants

View Source 
const (
	// WeightLimitPerMinute is the quota Binance publishes for REQUEST_WEIGHT
	// over a one-minute window, per IP address.
	WeightLimitPerMinute = 6000

	// KlinesWeight is what one /api/v3/klines call costs against that quota.
	KlinesWeight = 2

	// DefaultWeightPerSecond is the sustained rate this library allows itself,
	// in weight units per second.
	//
	// The quota works out to 100 weight per second. This takes 40 of them —
	// 20 klines calls a second, enough to page a two-day tail of 1s candles in
	// about nine seconds — and deliberately leaves the majority unspent.
	//
	// The headroom is not timidity. The quota is per IP, so anything else on
	// the machine draws from the same 6000: a live trading bot, another
	// backtest, a second copy of this library. Consuming the whole budget
	// because we are entitled to it is how a history download gets a trading
	// process banned, and the failure would appear in the trading process's
	// logs rather than in ours.
	DefaultWeightPerSecond = 40

	// DefaultBurst is how much weight may be spent instantaneously after an
	// idle period, in weight units.
	//
	// Ten klines calls. A burst exists so that a short pipeline is not paced
	// artificially — the first ten pages go out immediately and only then does
	// the sustained rate bind — and is kept small so that a saturated worker
	// pool cannot open with a spike large enough to matter.
	DefaultBurst = 20
)

Rate limiting constants, all measured rather than assumed. See the file comment for where the numbers come from.

View Source 
const ChecksumSuffix = ".CHECKSUM"

ChecksumSuffix is appended to an archive's key to get its sidecar's key. 

data/spot/monthly/klines/BTCUSDT/1h/BTCUSDT-1h-2024-01.zip
data/spot/monthly/klines/BTCUSDT/1h/BTCUSDT-1h-2024-01.zip.CHECKSUM
View Source 
const DefaultAPIBaseURL = "https://data-api.binance.vision"

DefaultAPIBaseURL is Binance's read-only market-data mirror.

It is the public half of the trading API — no key, no signature, market data only — and it is a genuinely different service from DefaultDownloadBaseURL despite the shared domain. In particular it enforces the trading API's rate limits, which the static bucket does not; see limiter.go.

View Source 
const DefaultBaseURL = "https://s3-ap-northeast-1.amazonaws.com/data.binance.vision"

DefaultBaseURL is the S3 REST endpoint for the bucket Binance publishes to.

Path-style rather than virtual-host style — the bucket name is the first path segment rather than a subdomain — because the bucket name contains dots, and a virtual-host URL would put those dots into the hostname where they would break TLS certificate matching.

View Source 
const DefaultDownloadBaseURL = "https://data.binance.vision"

DefaultDownloadBaseURL is where the archives themselves are served.

Note that this is Binance's own domain, not the S3 endpoint DefaultBaseURL uses. Fetching an object and listing objects genuinely go through different front doors — see the package comment — and only listing is tied to AWS. Downloads keep working through a CDN, a mirror or a migration.

View Source 
const KlineFields = 12

KlineFields is how many columns one kline row carries.

Twelve, the same twelve in the same order as the CSV inside an archive — verified against the live endpoint on 2026-08-20. That is what lets the root package feed a REST row and an archive row to one decoder instead of maintaining two that must agree.

View Source 
const MaxChecksumSize = 4 << 10

MaxChecksumSize bounds how much of a sidecar ReadChecksum will read. The real files are 91 bytes — 64 hex digits, two spaces and a file name — so anything approaching this is not a checksum and there is no reason to buffer it.

Exported so the cache's own tests can name the boundary they are crossing rather than repeating the number and hoping the two stay equal.

View Source 
const MaxKlinesLimit = 1000

MaxKlinesLimit is the largest page the endpoint will serve. Documented as "Default: 500; Maximum: 1000".

Variables

View Source 
var (
	// ErrNotFound reports a 404: the object is not there. For an archive this
	// is routine rather than exceptional — a month Binance has not published
	// yet, or a day before the symbol was listed — which is why it is a value
	// to branch on and not a failure.
	ErrNotFound = errors.New("object not found")

	// ErrRateLimited reports a 429 that survived the retry policy. Reaching
	// this means backing off inside one request was not enough and the caller
	// should slow the whole pipeline down.
	//
	// Errors carrying it are [*RateLimitError], which also reports how long the
	// server asked for; errors.Is finds this sentinel through that type's
	// Unwrap, so a caller that only wants to know "was it a 429?" needs no
	// knowledge of the struct.
	ErrRateLimited = errors.New("rate limited")
)

Errors this package reports as sentinels, for the root package to translate into its own. They exist because internal/vision cannot import the root — the root imports it — so binancedata.ErrNotAvailable is out of reach from here. The translation happens at the boundary, in the root's download.go.

View Source 
var ErrBadRequest = errors.New("request rejected")

ErrBadRequest reports a 4xx: the request was understood and refused. An unknown symbol and an unsupported interval both land here.

Retrying cannot help, which is why it is separate from the statuses [retryableStatus] lists. When the body carried Binance's own explanation the error is an *APIError naming the code it returned; when it did not, the status alone is the verdict. That distinction deliberately does not change the sentinel — whose fault a refusal is depends on the status class, not on whether the server chose to explain itself.

View Source 
var ErrIPBanned = errors.New("ip banned")

ErrIPBanned reports an HTTP 418: the address is banned, not merely throttled.

Binance escalates a 429 that the client keeps ignoring into this, and the ban runs from two minutes to three days depending on how often it has happened. It is kept distinct from ErrRateLimited because the correct response is different in kind — there is no backoff short enough to ride it out, and retrying is what earns the next, longer one. Errors carrying it are *RateLimitError, so a caller that only asks "should I slow down?" still finds ErrRateLimited through the same value.

View Source 
var ErrMalformedResponse = errors.New("malformed response")

ErrMalformedResponse reports that a 200 carried bytes this package could not read as a klines response: not JSON, not an array of arrays, a row with the wrong number of columns, or a column holding something no number can be recovered from.

It is the JSON counterpart of a corrupt archive, and the root package maps it onto exactly that sentinel. Without it the two halves of one condition part company: a price that will not parse as a decimal is caught by codec.go and reported as ErrCorruptArchive, while a body that is not JSON at all reaches the caller untyped — the same "Binance sent bytes this library cannot understand" arriving as two different answers depending on which layer noticed first.

View Source 
var ErrServerError = errors.New("server error")

ErrServerError reports a 5xx: Binance's side failed, and nothing about the request needs changing.

It exists because the alternative is worse than having no sentinel at all. [retryableStatus] already retries 500, 502, 503 and 504, and when those attempts are exhausted [doWithRetry] hands back the response so the caller can report what the server actually said. Binance answers a 5xx with the same {"code","msg"} document it uses for a 400 — {"code":-1001,"msg":"Internal error; unable to process your request."} — so a status-blind reading of the body reports an outage as the caller's own bug, and the root package's ErrInvalidRequest documents itself as "always the caller's to fix". A worker pool told that would refuse to retry or fall back precisely when both would have worked.

It is not translated into a root-package sentinel. The public vocabulary in errors.go says nothing about server-side failure, and an error that arrives unrecognised is a smaller lie than one that arrives mislabelled.

Functions

func BurstFor 

func BurstFor(ratePerSecond float64) int

BurstFor returns the burst that pairs with a sustained rate, in weight units.

The rule is half a second's worth of the rate, never less than one klines call. At DefaultWeightPerSecond that is exactly DefaultBurst, which is the point: the shipped policy is not a pair of unrelated constants but one number and a rule, so a caller who lowers the rate gets a proportionally smaller spike rather than the shipped burst on top of their slower refill.

The floor matters more than it looks. rate.Limiter.WaitN returns an error rather than waiting when n exceeds the burst, so a bucket smaller than KlinesWeight would fail every klines call forever instead of pacing it — the failure mode a caller asking for a very low rate is least expecting.

func FormatChecksum 

func FormatChecksum(sum, name string) string

FormatChecksum renders a sidecar in the form Binance publishes: the hash, two spaces, the archive's file name, and no trailing newline.

The cache writes one of these beside every archive it stores, so that tier 1 on disk is the pair of files Binance served rather than the archive plus a hash in some format of this project's own invention. Two consequences follow from matching the published format byte for byte: `sha256sum -c` verifies a cache directory with no tooling from here, and the round trip through ParseChecksum is exercised against the real fixtures rather than against a convention this package agreed with itself.

It lives next to the parser deliberately. A writer and a reader of the same format in two different files is how they drift.

func NewHTTPClient 

func NewHTTPClient() *http.Client

NewHTTPClient returns the *http.Client this library uses for every request in a process.

One client, not one per call

An http.Client is not a connection. It is a handle onto a Transport, and the Transport is what holds the pool of live TCP connections. Creating a client per request therefore creates a pool per request, uses one connection from it, and throws the pool away — so every single download pays a fresh TCP handshake and a fresh TLS handshake, roughly 100–200 ms of round trips before a byte of payload moves. Over a few hundred archives that is minutes spent re-introducing ourselves to a server we never left.

That is bug 8 of the ten this library exists to fix, and it is the reason both NewLister and NewDownloader take a client rather than making one: sharing is only possible if the layer above owns it.

http.Client is safe for concurrent use by multiple goroutines — that is a documented guarantee, not an accident — so one value serves the whole worker pool.

func NewLimiter 

func NewLimiter(ratePerSecond float64, burst int) *rate.Limiter

NewLimiter returns a limiter allowing ratePerSecond weight per second, with a bucket holding burst weight.

A non-positive rate or burst is replaced by the default, so the zero-ish call NewLimiter(0, 0) yields the standard policy rather than a limiter that blocks forever — which is the shape of mistake that only shows up in production. rate.NewLimiter itself has no such guard: a zero rate is rate.Limit(0), which refills nothing, and a zero burst rejects every call outright.

It returns *rate.Limiter rather than a wrapper type. A wrapper would exist only to rename WaitN, and this package's own constants already carry the domain meaning: the unit is weight, and one klines call is KlinesWeight of it.

func ParseChecksum 

func ParseChecksum(b []byte, wantName string) (string, error)

ParseChecksum extracts the hash from a .CHECKSUM sidecar.

The format is the one `sha256sum` writes, which the real files follow exactly — 64 hex digits, two spaces, the file name, and, in the files Binance publishes, no trailing newline: 

ea666c96c34414515 ... 4d99  BTCUSDT-1h-2024-01-15.zip

The parse accepts more than that: any run of whitespace as the separator, an optional trailing newline, and coreutils' "*" binary-mode marker before the name. Being liberal about whitespace costs nothing and survives a future where the sidecar gains a newline. Being liberal about the *hash* would cost everything, so its length and alphabet are checked exactly.

wantName, when non-empty, must match the name the sidecar carries.

It is exported for the cache, which reads sidecars back off disk rather than off the network. Two parsers for one format is one parser that gets fixed and one that does not, so the local path and the network path share this.

func ReadChecksum 

func ReadChecksum(r io.Reader, wantName string) (string, error)

ReadChecksum reads a .CHECKSUM sidecar from r and returns the hash in it.

It is the bounded read that has to happen before ParseChecksum can be handed a byte slice, and it is exported for the same reason the parser is: the cache reads sidecars off disk while this package reads them off the network, and the *reading* half of that has the same two decisions in it as the parsing half. Keeping them together means the size limit is one constant and the "too large" error is one sentence, rather than two of each drifting apart in two packages.

The caller supplies the context — a URL here, a file name in the cache — by wrapping what comes back, so nothing in this function has to know which it was reading.

Types

type API 

type API struct {
	// contains filtered or unexported fields
}

API reads klines from the REST mirror.

Like Lister and Downloader it holds its client rather than making one, so that every request in the process shares a connection pool. Unlike them it is paced, for the reason limiter.go opens with: this endpoint has a quota and the bucket does not.

The limiter is not a field here. It is captured by the Reserve closure on the policy, which is the only thing that reads it — a second reference on the struct would be state that looks consultable and never is.

func NewAPI 

func NewAPI(baseURL string, client *http.Client, p Policy, lim *rate.Limiter) *API

NewAPI returns an API reading from baseURL using client, retrying per p and pacing itself with lim.

An empty baseURL means DefaultAPIBaseURL, a nil client means the process-wide client from NewHTTPClient, the zero Policy means DefaultPolicy, and a nil limiter means the process-wide limiter — which is the one default that is load-bearing rather than convenient. The quota is per IP address, so two APIs each pacing themselves correctly still exceed it together; sharing one limiter is what makes the accounting add up.

func (*API) Klines 

func (a *API) Klines(ctx context.Context, q KlineQuery) (KlinesPage, error)

Klines fetches one page.

The rate limiter is not consulted here. It is installed on the policy by NewAPI and spent inside [doWithRetry], once per HTTP request rather than once per call — the difference being that a call which retries makes several, and does so exactly when the quota is under pressure. See Policy.Reserve.

The validation below runs first, so a query that could never be sent costs no budget: the cheapest request is the one refused before it exists.

type APIError 

type APIError struct {
	// Status is the HTTP status line, kept because a 400 and a 403 with the
	// same body mean different things.
	Status string

	// StatusCode is the same status as a number, and it decides which sentinel
	// [APIError.Unwrap] reports. Binance uses one body shape for refusals and
	// for its own failures, so the status class is the only thing in the
	// response that says whose fault it is.
	StatusCode int

	// Code and Msg are Binance's own. Code is negative in every documented
	// case; a zero Code means the body did not parse as an error document.
	Code int
	Msg  string
}

APIError is a refusal the REST API described in its body, in the shape it documents: 

{"code":-1121,"msg":"Invalid symbol."}

Verified against the live endpoint on 2026-08-20, which answers exactly that with HTTP 400 for a symbol that does not exist.

The code is carried rather than interpreted. This package has no way to know whether -1121 means the caller typed the symbol wrong or the pair was delisted last week, and inventing a distinction here would put a guess where the root package can make an informed decision.

func (*APIError) Error 

func (e *APIError) Error() string

func (*APIError) Unwrap 

func (e *APIError) Unwrap() error

Unwrap is what makes errors.Is(err, ErrBadRequest) true for this type — or errors.Is(err, ErrServerError), for the statuses where the failure is Binance's rather than ours.

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader fetches single objects out of the bucket.

Like Lister it holds a client rather than creating one, so that every request in the process shares one connection pool. The two types are separate because they talk to different hosts, and collapsing them would hide that.

func NewDownloader 

func NewDownloader(baseURL string, client *http.Client, p Policy) *Downloader

NewDownloader returns a Downloader reading from baseURL using client.

An empty baseURL means DefaultDownloadBaseURL, a nil client means the process-wide client from NewHTTPClient, and the zero Policy means DefaultPolicy. Every test in this package aims baseURL at an httptest.Server.

The nil client deliberately does not mean http.DefaultClient: that would hand back a transport keeping two idle connections per host, which is the very bug correctness requirement 8 names. See [defaultClient].

func (*Downloader) Checksum 

func (d *Downloader) Checksum(ctx context.Context, archiveKey string) (string, error)

Checksum fetches the .CHECKSUM sidecar for archiveKey and returns the SHA-256 it contains, in lowercase hex.

archiveKey is the archive's own key; the suffix is appended here so that no caller has to remember it, and so the sidecar's file name can be checked against the archive it claims to describe. A sidecar naming a different file means a misrouted request or a mirror serving stale content, and trusting it would compare an archive against another archive's hash — which fails as a checksum mismatch and sends whoever investigates hunting for a corruption that never happened.

func (*Downloader) Download 

func (d *Downloader) Download(ctx context.Context, key string, dst io.Writer) (Result, error)

Download writes the object at key into dst and returns its SHA-256 and size.

The hash is computed in the same pass as the write

io.MultiWriter feeds each block of the body to both the destination and the hasher, so the bytes are hashed as they stream past rather than by reading the file back afterwards. A 93 MB archive is therefore hashed for free, in terms of both I/O and memory: nothing is buffered, and the peak footprint is one 32 KiB copy buffer regardless of how large the archive is. This is what makes verifying every download affordable enough to be non-optional.

What dst holds after an error

Whatever arrived before the failure. A partially written destination is unavoidable when streaming — the alternative is buffering the whole archive in memory to keep the write atomic, which is the cost this design exists to avoid — so the contract is that the caller discards dst unless Download returned nil. That is no burden on the only caller that matters: the cache writes to a temporary file and renames it into place only on success, so a failed download leaves a temp file it was going to delete anyway.

Retries inside this call therefore cover the request and status phases, which is where nearly all failures happen. A connection that dies mid-body is reported to the caller rather than restarted, because restarting would need to rewind dst and an io.Writer cannot be rewound.

type KlineQuery 

type KlineQuery struct {
	// Symbol is the pair in Binance's own spelling, "BTCUSDT". It is not
	// normalised here; the root package does that before building a query.
	Symbol string

	// Interval is the REST spelling of the interval, which differs from the
	// archive spelling for exactly one value: a month is "1M" here and "1mo"
	// in a bucket path. See binancedata.Interval.RESTParam.
	Interval string

	// Start and End bound the page. A zero End means "no upper bound", which
	// the endpoint reads as "up to the present".
	Start time.Time
	End   time.Time

	// Limit is how many rows to return, at most [MaxKlinesLimit]. Zero means
	// the endpoint's own default of 500.
	Limit int
}

KlineQuery is one page of klines to ask for.

Start and End are half-open, [Start, End), matching the rest of this project — and deliberately not matching the endpoint, whose startTime and endTime are both inclusive. The conversion happens in one place, in API.Klines, rather than at every call site where a forgotten adjustment would duplicate or drop a single candle at the seam.

type KlinesPage 

type KlinesPage struct {
	// Klines are the rows, in the order the endpoint sent them — which is
	// ascending by open time, though nothing here relies on that. The root
	// package verifies the ordering itself, because a page that repeated or
	// reversed a candle would otherwise stall the pagination loop.
	Klines []RawKline

	// UsedWeight is the X-MBX-USED-WEIGHT-1M header: how much of the
	// [WeightLimitPerMinute] quota this IP has spent in the current minute,
	// counting this request. Zero when the header was absent or unreadable.
	//
	// It is decoded rather than acted on. The limiter's job is to keep this
	// number low, and this is the only way to notice when that is not working
	// — a second process on the same address spending the same quota, most
	// likely, which no amount of local accounting can detect.
	//
	// The root package's restapi.go reports it to the configured slog.Logger:
	// at debug level for every page, and once per fetch at warn level past four
	// fifths of [WeightLimitPerMinute]. Reporting rather than reacting is the
	// division of labour this package keeps everywhere — a page knows what one
	// request cost, and only the layer owning the pipeline can decide that the
	// pipeline should slow down.
	UsedWeight int
}

KlinesPage is what one call returned.

type Lister 

type Lister struct {
	// contains filtered or unexported fields
}

Lister reads object listings from the bucket.

It holds an *http.Client rather than creating one, because connection reuse only happens if the same client is used for every request in the process — a fresh client per call reopens the TLS connection each time, which is the eighth bug in the implementation this library replaces.

func NewLister 

func NewLister(baseURL string, client *http.Client, p Policy) *Lister

NewLister returns a Lister reading from baseURL using client.

An empty baseURL means DefaultBaseURL, a nil client means the process-wide client from NewHTTPClient, and the zero Policy means DefaultPolicy. Accepting zero values for all three keeps tests short while leaving the production wiring explicit.

The nil client deliberately does not mean http.DefaultClient — see [defaultClient] for why that default would quietly reintroduce the connection churn this package exists to remove.

Listings are retried on the same policy as downloads, and it matters more here than it looks: a listing that fails is a listing that cannot be read as empty, so every transient 503 that is not retried becomes a request that fails outright rather than one that costs an extra half second.

func (*Lister) List 

func (l *Lister) List(ctx context.Context, prefix, startAfter string) ([]Object, error)

List returns every object directly under prefix, in lexicographic order.

The three outcomes

Go's two-value return already provides the three states this needs, and the distinction is the whole reason this function exists: 

objects, nil    the bucket answered; these are the objects (possibly none)
nil, err        the bucket did not answer; nothing is known

The trap is the first case with an empty slice. Asking for a symbol that does not exist returns HTTP 200 and a well-formed listing with no Contents at all — verified 2026-08-18 against prefix .../NOSUCHSYM/1h/. There is no 404. So "this symbol was never listed" and "this interval has no archives yet" and "you typed the prefix wrong" are one indistinguishable answer, and it looks exactly like success.

What callers must not do is collapse that into the error case, or treat a failed listing as an empty one. An empty listing means "Binance has nothing here", which is a fact worth acting on; an error means "we do not know", which is never a reason to return zero candles and a nil error. That conflation is how the ported implementation lost days at the end of a range.

startAfter

Listings resume rather than restart. Keys sort lexicographically and Binance names archives with ISO dates — BTCUSDT-1m-2024-06-01.zip — so lexicographic order is chronological order, and passing the key a range begins at seeks straight to it. Without that, listing the daily archives of a symbol listed in 2017 costs seven round trips before reaching 2024; with it, one.

Pass the empty string to list from the beginning.

type Object 

type Object struct {
	Key  string
	Size int64
}

Object is one entry from a listing: the object's full key and its size.

Size is carried because it is free — S3 sends it whether or not it is wanted — and because it is the cheapest way to notice a placeholder file. Binance has published archives that are a few hundred bytes of nothing.

type Policy 

type Policy struct {
	// MaxAttempts is the total number of tries, not the number of retries. 1
	// means "no retrying"; 0 means "use the default".
	MaxAttempts int

	// BaseDelay is the first backoff interval. Each subsequent attempt doubles
	// it, up to MaxDelay.
	BaseDelay time.Duration

	// MaxDelay caps the exponential growth. Without a cap, attempt eight of an
	// eight-attempt policy would wait over a minute.
	MaxDelay time.Duration

	// Jitter maps the computed delay ceiling to the delay actually waited.
	//
	// This is not decoration. Without it, a bounded pool that starts forty
	// downloads at once and receives forty 503s retries all forty at exactly
	// the same instant, and keeps doing so in lockstep — a thundering herd
	// that recreates the overload it is backing off from. The default is
	// "full jitter": a uniform draw from [0, ceiling), which spreads the herd
	// across the whole window.
	//
	// Tests set this to the identity function to make delays exact.
	Jitter func(time.Duration) time.Duration

	// After is the timer. It exists as a field so that tests can advance time
	// without spending it: the suite asserts on the delays that were requested
	// rather than sleeping through them.
	//
	// Defaulting to time.After is safe from Go 1.23 onward — before that, the
	// timer it created stayed alive until it fired, so an abandoned time.After
	// in a select leaked until its deadline passed. This module's floor is
	// 1.24, so the timer is garbage-collected as soon as nothing references it.
	After func(time.Duration) <-chan time.Time

	// Now reads the wall clock, and is used for exactly one thing: converting
	// an HTTP-date Retry-After header into a duration. It is injected for the
	// same reason as After — a test asserting on a date-form Retry-After must
	// not depend on what time it is when the suite runs.
	Now func() time.Time

	// Reserve, when set, is called before every attempt and must not return
	// until the request it is about to permit fits inside whatever budget it
	// is keeping. Nil means unmetered, which is what the bucket endpoints use:
	// a static file server publishes no quota.
	//
	// # Why the reservation lives here rather than at the call site
	//
	// Because the thing being counted is a request, and this is the only
	// function that knows how many of those an attempt-carrying call makes.
	// [API.Klines] reserved once, before calling in, and the number looked
	// right — one call, one reservation — right up until the endpoint started
	// failing. A retryable status turns one reservation into as many as
	// MaxAttempts requests, and it does so precisely when the budget matters
	// most, because 429 and every retryable 5xx are the statuses that trigger
	// the extra attempts. The limiter that exists to pre-empt an IP ban was
	// what permitted the burst that earns one.
	//
	// # And why this is not a second backoff
	//
	// The obvious objection to reserving before a *retry* is that the attempt
	// is already waiting out its own delay, so a limiter wait on top of it
	// applies two delays for one problem. It does not, in the case that
	// matters: the reservation returns immediately whenever the budget has
	// room, which after several hundred milliseconds of backoff it normally
	// does. When it does not have room, waiting is not a duplicated penalty —
	// it is the pipeline being over budget, which is the one condition this
	// hook exists to notice.
	Reserve func(context.Context) error
}

Policy is how [doWithRetry] behaves: how many attempts, how long between them, and — for tests — where the clock and the randomness come from.

The zero Policy is not usable directly; every function taking one calls [Policy.withDefaults] first, so a caller that wants the defaults can pass Policy{} and a caller that wants to change one field can set just that one. This is the same convention NewLister uses for its base URL and client, and it is the lightweight cousin of the functional-options pattern the public API uses: fine for an internal struct, too rigid for an exported one.

func DefaultPolicy 

func DefaultPolicy() Policy

DefaultPolicy is the retry behaviour used when a caller passes the zero Policy: four attempts, 500 ms doubling to a 8 s ceiling, full jitter.

Four attempts with these delays spans at most 500 ms + 1 s + 2 s ≈ 3.5 s of waiting, which is long enough to ride out a load balancer restarting and short enough that a genuinely dead endpoint is reported while someone is still watching the terminal.

type RateLimitError 

type RateLimitError struct {
	// Key is the object whose request was throttled.
	Key string

	// RetryAfter is what the server's Retry-After header asked for, or 0 when
	// it sent none or sent one that had already elapsed.
	//
	// It is reported exactly as received, not clamped. maxRetryAfter bounds
	// what this package will itself wait inside one request; how long a whole
	// pipeline should pause is the pool's policy to set, and quietly rounding a
	// misconfigured proxy's 24 hours down to 30 seconds here would hide the
	// misconfiguration rather than surface it. Callers should bound it.
	RetryAfter time.Duration

	// Banned distinguishes an HTTP 418 from an HTTP 429: the address is barred
	// rather than merely being asked to slow down.
	//
	// Only the REST API produces it — the static bucket has no quota — and it
	// is the escalation Binance applies to a client that keeps ignoring 429s.
	// The ban runs from two minutes to three days and lengthens with repeat
	// offences, so the right response is to stop rather than to back off: a
	// retry cannot outlast it and does earn the next one. See klines.go.
	Banned bool
}

RateLimitError is a 429 that outlived the retry policy, together with the server's own estimate of when it will be ready again.

Why the duration needs a type

The retry loop reads Retry-After to schedule its own backoff and then throws the value away, because within one request it has no further use for it. But a 429 that survives every attempt is not a fact about one request — it is the pipeline as a whole going too fast, and the layer that owns the worker pool is the only one that can slow it down. Handing that layer a bare "rate limited" tells it to back off without telling it for how long, so the number the server actually supplied would have to be guessed at one frame above where it arrived.

Reading it

A caller that only branches on the condition uses errors.Is and never sees this type. A caller that wants the duration uses errors.As: 

var rl *vision.RateLimitError
if errors.As(err, &rl) && rl.RetryAfter > 0 {
    pool.PauseFor(rl.RetryAfter)
}

The pointer receiver on Error is why errors.As wants **RateLimitError: the method set containing Error belongs to *RateLimitError, so that — not the bare struct — is what implements error.

func (*RateLimitError) Error 

func (e *RateLimitError) Error() string

func (*RateLimitError) Unwrap 

func (e *RateLimitError) Unwrap() []error

Unwrap is what makes errors.Is(err, ErrRateLimited) true for this type. Returning the sentinel rather than embedding it keeps the two ways of asking — the condition and the detail — from needing to agree about anything beyond this one method.

Why it returns a slice

A ban is two facts at once: the address is barred, and the pipeline is going too fast. A caller that only asks "should I slow down?" should get a yes from a 418 as readily as from a 429, while one that can tell the difference should be able to. Since Go 1.20 an Unwrap returning []error attaches both, and errors.Is walks every branch — so the coarse question and the precise one are answered by the same value without either having to know about the other.

type RawKline 

type RawKline [KlineFields]string

RawKline is one row exactly as Binance sent it: twelve fields, every one of them the original characters.

An array rather than a slice, so the length is part of the type and a short row cannot be constructed by accident. Numeric columns hold their literal text — "1704067200000", "47134" — and quoted columns hold their contents with the quotes and any escapes resolved.

type Result 

type Result struct {
	// SHA256 is the hash of the bytes as they were received, in lowercase hex
	// — the same form Binance writes into the .CHECKSUM sidecar, so the two
	// compare with ==.
	SHA256 string

	// Size is how many bytes were written to the destination.
	Size int64
}

Result describes what one download produced.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.