binancedata

README

binance-data-downloader

Historical Binance candles, downloaded once and cached properly. A Go library, plus a bmd command-line tool built on it.

Ask for a date range. You get back every candle in it, verified against Binance's published checksums. Ask again tomorrow and it comes from disk — no network, no re-parsing. A backtest can re-read five years of one-minute data on every run without paying for it twice.

Binance splits its history across two sources: bulk ZIP archives that lag about a day behind, and a REST endpoint for everything newer. This library stitches them together so you never have to think about the seam.

Scope: spot klines only. Not futures, not order books, not trades.

Install

Requires Go 1.25 or newer.

# As a library
go get github.com/algo-one/binance-data-downloader

# As a CLI
go install github.com/algo-one/binance-data-downloader/cmd/bmd@latest

Try it

bmd download -symbol BTC/USDT -interval 1h -start 2024-01-01 -end 2024-03-31

That writes a CSV to the current directory. Run it a second time and it finishes almost instantly — everything is already cached.

The other commands:

bmd list   -symbol BTC/USDT -interval 1mo      # what Binance actually publishes
bmd cache                                      # what your cache holds
bmd prune                                      # reclaim disk; -n to preview
bmd evict  -symbol BTC/USDT -before 2023-01-01 # delete data you are done with
bmd verify                                     # re-hash the cache

Output is csv, json or parquet. Full flag reference in docs/cli.md, or run bmd help.

Use it from Go

// No options needed — the cache lands in your OS cache directory.
loader, err := binancedata.NewLoader()
if err != nil {
    return err
}

klines, err := loader.Fetch(ctx, binancedata.Request{
    Symbol:   "BTC/USDT",
    Interval: binancedata.Interval1h,
    Market:   binancedata.MarketSpot,
    Start:    start,
    End:      end, // leave zero for "now, at call time"
})

Each Kline carries open/high/low/close, both volumes, the taker-buy split and the trade count.

Two variants for bigger jobs:

• loader.Stream(ctx, req) yields candles one at a time, for ranges too large to hold in memory.
• loader.FetchAll(ctx, reqs) runs several requests under one concurrency budget and downloads shared archives only once.

Things worth knowing

Ranges are inclusive at both ends. A candle comes back when Start <= OpenTime <= End. So a full year of 2024 is Start 2024-01-01 and End 2024-12-31T23:59:59.999999999Z. Read End as the open time of the last candle you want — writing End 2025-01-01 is legal, but it asks for the candle opening on New Year's Day and costs you January's archive to fetch.

Batch your symbols, don't batch your processes. Pass lists — -symbol BTC/USDT,ETH/USDT -interval 1m,1h — rather than running one bmd per pair. Binance rate-limits per IP address, and the limiter enforcing it lives in one process, so parallel processes will blow through the limit between them. Every pair gets its own output file.

Moving the cache. export BMD_CACHE_DIR=/mnt/big-disk/bmd redirects every command that takes -cache-dir; the flag wins if both are set. The library reads no environment variables at all — Go callers pass binancedata.WithCacheDir instead, so an unrelated env var can never redirect where your program writes.

How it works

Numbers are exact. Prices and volumes are udecimal.Decimal, never float64. Binance quote volumes reach 20 significant digits; float64 holds 15.95. No int64 fixed-point scale saves you either — real PEPE daily volume overflows int64 at 1e8 scaling, silently and negatively. The archives are text, the numbers in them are exact, and they stay that way. Details in docs/numbers.md.

The cache has two tiers. Tier 1 is the raw ZIP exactly as Binance served it, next to its .CHECKSUM. Tier 2 is a Parquet file derived from it — that's what reads actually hit. Each Parquet stores the SHA-256 of the archive it came from in its own footer, so a cached file is trusted without re-hashing anything. In the steady state nothing is ever rebuilt. Details in docs/caching.md.

Nothing is evicted automatically. File timestamps record when data was downloaded, not when it was used, so there's no honest expiry rule to apply. You decide: bmd prune drops archives that reads no longer need (~40% of the cache), bmd evict removes entries you name.

Library first. This is built to sit inside a backtesting framework. The CLI is a thin shell over the library, not the other way round — anything bmd can do, your Go code can do.

Documentation

Start with the API reference on pkg.go.dev, or read the same thing offline with go doc github.com/algo-one/binance-data-downloader.

example_test.go holds sixteen worked examples and is probably the fastest way in. Seven of them run on every test run with their output checked, so they can't quietly go stale.

Longer form:

Document	Contents
docs/architecture.md	How the pieces fit together, and the staged build plan
docs/caching.md	The two-tier cache, its invariants, and why it exists
docs/cli.md	The bmd command-line tool
docs/numbers.md	Why prices are decimals, and what the alternatives measured
docs/go-notes.md	The Go idioms this codebase leans on, in one place

The code is commented far more heavily than typical Go. That's deliberate — this repository doubles as a way to learn the language, so comments explain why a construct is used, not just what it does.

Development

Tooling is managed with mise, which pins the Go version, the linter and the test runner so every machine and CI runner use identical bits.

mise trust      # once, to allow this repo's mise.toml
mise install    # fetch the pinned toolchain
mise tasks      # list everything below

Task	What it does
mise run build	Compile the bmd CLI into ./bin
mise run test	Run all tests with the race detector
mise run lint	Run golangci-lint
mise run fmt	Format all Go code in place
mise run fmt:check	Fail if anything is unformatted (CI uses this)
mise run cover	Test with coverage and open the HTML report
mise run tidy	Sync go.mod with the imports in the code
mise run audit	Check dependencies against the Go vulnerability database
mise run release:snapshot	Build the release artefacts locally, publishing nothing
mise run ci	Everything CI runs, in order

No test in this repository touches Binance. Network paths run against httptest servers with committed fixtures.

Versioning

v0.x. Everything documented here works and is covered by tests, but the API can still change between minor versions — v0.2.0 may rename something v0.1.0 used. Pin an exact version if that matters to you.

License

BSD 3-Clause. See LICENSE.

Documentation

Overview

Package binancedata downloads and caches historical Binance market data.

It is both an embeddable Go library and the engine behind the bmd command-line tool. The design goal is that a backtest can ask for five years of candles and get them back quickly on every run, without re-downloading or re-parsing anything it has already seen.

Getting started

Build a Loader once and share it — one per process. It is safe for concurrent use, and the concurrency limit, the connection pool and the REST rate limiter all live on it, so two Loaders each pacing themselves correctly would still exceed Binance's per-IP quota together.

loader, err := binancedata.NewLoader()
if err != nil {
    return err
}

klines, err := loader.Fetch(ctx, binancedata.Request{
    Symbol:   "BTC/USDT",
    Interval: binancedata.Interval1h,
    Market:   binancedata.MarketSpot,
    Start:    start,
    End:      end, // leave zero for "now, at call time"
})

Ranges are closed: a candle is returned when Start <= OpenTime <= End, so a full year of 2024 is Start 2024-01-01 and End 2024-12-31T23:59:59.999999999Z. See Request for why the last instant is spelt out that way, and for what End 2025-01-01 would fetch instead.

Loader.Stream yields the same candles one at a time for a range too large to hold at once, and Loader.FetchAll runs several requests under one concurrency budget, downloading whatever they have in common exactly once.

Five more calls answer questions that are not "give me candles". Loader.Available reports what Binance actually publishes for a symbol and interval, holes included — the archives have them, and no calendar predicts which. Loader.VerifyCache re-hashes cached archives against the checksums they were published with. Loader.CacheUsage measures what the cache holds, Loader.PruneArchives reclaims the part of it that reads no longer need, and Loader.EvictCache removes entries themselves when a window has moved on. WriteParquet writes candles in the same format the cache stores its second tier in, for a query engine to read.

Examples

example_test.go carries a worked example for most of the surface above, and they are the fastest way in. Seven of them execute on every test run with their printed output checked — the pure ones, which need no network — and the rest are compiled but not run, because running them would mean fetching from Binance and no test in this repository does that. Both kinds are checked by the compiler, so an example naming a field that no longer exists fails the build rather than misleading somebody.

Stability

The version is v0.x. Everything documented here works and is tested, but the API carries no compatibility promise yet: a v0.2.0 may rename something a v0.1.0 caller used. Pin an exact version if that matters.

Where the data comes from

Binance publishes bulk archives at https://data.binance.vision as monthly and daily ZIP files, each accompanied by a .CHECKSUM sidecar holding a SHA-256 of the archive. Archives lag real time by roughly a day, so the most recent candles are fetched instead from the REST mirror at https://data-api.binance.vision. This package hides that split: you ask for a time range, and it works out which sources cover it.

Caching

Two tiers live under the cache directory:

• Tier 1 is the raw ZIP exactly as Binance served it, plus its .CHECKSUM. It is the source of truth and is never re-downloaded once verified.
• Tier 2 is a Parquet file derived from tier 1, which is what reads actually hit. Parsing a month of one-minute candles out of CSV costs tens of milliseconds; reading the Parquet costs a fraction of that.

Tier 2 records the SHA-256 of the tier-1 archive it was built from in its Parquet footer, so a cached Parquet can be trusted without re-hashing the ZIP. In the steady state nothing is ever rebuilt. The caching.md document, linked under Documentation below, gives the full invariants.

A read never opens tier 1, so tier 1 can be deleted to reclaim disk — about 40% of a cache, since the Parquet is the larger of the two files. That is what Loader.PruneArchives does, and it costs a download rather than a decode on the one path that still needs an archive: rebuilding.

The other 60% is the data. Loader.EvictCache removes whole entries — archive, sidecar and Parquet — selected by symbol, interval or the period they cover, which is the one operation here that a later read pays for in full. The cache never evicts on its own: file times say when an entry was downloaded rather than when it was used, so there is no honest expiry rule to apply without recording every read.

A note on numbers

Prices and volumes are github.com/quagmt/udecimal.Decimal, not float64. Binance quote volumes reach 20 significant digits, which float64 (15.95 digits) cannot represent exactly, and no int64 fixed-point scale covers the range either — real meme-coin volumes overflow int64 at 1e8 scaling. The archives are text, the values in them are exact, and this package preserves them digit for digit.

Convert to float64 explicitly at the point where you need it, which for most callers is when feeding an indicator library.

Documentation

The repository carries the longer-form material, one file per concern:

• https://github.com/algo-one/binance-data-downloader/blob/main/docs/architecture.md for how the pieces fit together
• https://github.com/algo-one/binance-data-downloader/blob/main/docs/caching.md for the cache design and its invariants
• https://github.com/algo-one/binance-data-downloader/blob/main/docs/numbers.md for why prices are decimals and what the alternatives measured
• https://github.com/algo-one/binance-data-downloader/blob/main/docs/cli.md for the bmd command-line tool
• https://github.com/algo-one/binance-data-downloader/blob/main/docs/go-notes.md for the Go idioms this codebase leans on

Example

Example is the whole library in one call: build a loader, ask for a range, get candles.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	// One Loader per process, built once and shared. The concurrency limit,
	// the connection pool and the REST rate limiter all live on it, so two
	// loaders each pacing themselves correctly would still exceed Binance's
	// per-IP budget together.
	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	klines, err := loader.Fetch(ctx, binancedata.Request{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1h,
		Market:   binancedata.MarketSpot,
		Start:    time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC),
		// The range is closed, so End is the open time of the last candle
		// wanted. Writing 2024-02-01 here would be legal and would ask for one
		// more candle — the one opening at midnight on the 1st — which costs
		// February's archive to fetch.
		End: time.Date(2024, 1, 31, 23, 0, 0, 0, time.UTC),
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(len(klines), "candles")
}

Example (Errors)

Example_errors shows the six sentinels and what each one means you should do.

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	_, err = loader.Fetch(ctx, binancedata.Request{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1h,
		Market:   binancedata.MarketSpot,
		Start:    time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC),
		End:      time.Date(2024, 1, 31, 23, 0, 0, 0, time.UTC),
	})

	// Errors are compared with errors.Is, never with ==. Every error this
	// package returns is wrapped at least once on its way out — with the URL,
	// the archive name, the row number — so the sentinel is somewhere in the
	// chain rather than at the end of it, and == would miss all of them.
	switch {
	case err == nil:
		fmt.Println("got the candles")

	case errors.Is(err, binancedata.ErrIPBanned):
		// Checked before ErrRateLimited, because a 418 carries both. There is
		// no backoff short enough to ride out a ban — it lasts from two
		// minutes to three days — and retrying is what earns the next, longer
		// one. Stop.
		log.Fatal("banned; stop the pipeline: ", err)

	case errors.Is(err, binancedata.ErrRateLimited):
		// Wait and try again. This is the one failure where waiting is the
		// right answer.
		log.Print("slow down: ", err)

	case errors.Is(err, binancedata.ErrInvalidRequest):
		// The request itself is wrong, and no network round trip was spent
		// finding out. Retrying is pointless; fix the request.
		log.Fatal("bad request: ", err)

	case errors.Is(err, binancedata.ErrNotAvailable):
		// Binance does not have this data: a symbol not yet listed on the
		// date asked for, or a day too recent to be published. A fact about
		// the world, and routinely the correct answer.
		log.Print("no data: ", err)

	case errors.Is(err, binancedata.ErrChecksum):
		// The bytes on disk or on the wire are not what Binance published.
		// Transient — the cache discards the file and the next fetch
		// re-downloads it.
		log.Print("corrupted transfer, worth retrying: ", err)

	case errors.Is(err, binancedata.ErrCorruptArchive):
		// Bytes that passed the checksum and still could not be parsed, which
		// means Binance published something this decoder does not understand.
		// Retrying produces the same bytes; this one is a bug report.
		log.Fatal("unparseable archive: ", err)

	default:
		log.Fatal(err)
	}
}

Index

• Constants
• Variables
• func Closes(klines []Kline) []float64
• func Highs(klines []Kline) []float64
• func Lows(klines []Kline) []float64
• func NormalizeSymbol(s string) (string, error)
• func OpenTimes(klines []Kline) []time.Time
• func Opens(klines []Kline) []float64
• func Version() string
• func Volumes(klines []Kline) []float64
• func WriteParquet(ctx context.Context, w io.Writer, seq iter.Seq2[Kline, error]) (int, error)
• type Availability
  • func (a Availability) DailyGaps() []time.Time
  • func (a Availability) MonthlyGaps() []time.Time
• type AvailabilityQuery
• type CacheEntry
• type CacheUsage
  • func (u CacheUsage) Total() int64
• type EvictOptions
• type EvictResult
• type Interval
  • func Intervals() []Interval
  • func ParseInterval(s string) (Interval, error)
  • func (i Interval) Duration() (time.Duration, bool)
  • func (i Interval) HasDailyArchives() bool
  • func (i Interval) HasMonthlyArchives() bool
  • func (i Interval) IsValid() bool
  • func (i Interval) MarshalText() ([]byte, error)
  • func (i Interval) RESTParam() string
  • func (i Interval) String() string
  • func (i *Interval) UnmarshalText(text []byte) error
• type Kline
  • func (k Kline) Equal(other Kline) bool
• type Loader
  • func NewLoader(opts ...Option) (*Loader, error)
  • func (l *Loader) Available(ctx context.Context, q AvailabilityQuery) (Availability, error)
  • func (l *Loader) CacheUsage(ctx context.Context) (CacheUsage, error)
  • func (l *Loader) EvictCache(ctx context.Context, opts EvictOptions) iter.Seq2[EvictResult, error]
  • func (l *Loader) Fetch(ctx context.Context, req Request) ([]Kline, error)
  • func (l *Loader) FetchAll(ctx context.Context, reqs []Request) (map[Request][]Kline, error)
  • func (l *Loader) PruneArchives(ctx context.Context, opts PruneOptions) iter.Seq2[PruneResult, error]
  • func (l *Loader) Stream(ctx context.Context, req Request) iter.Seq2[Kline, error]
  • func (l *Loader) VerifyCache(ctx context.Context) iter.Seq2[CacheEntry, error]
• type Market
  • func ParseMarket(s string) (Market, error)
  • func (m Market) IsValid() bool
  • func (m Market) MarshalText() ([]byte, error)
  • func (m Market) String() string
  • func (m *Market) UnmarshalText(text []byte) error
• type Option
  • func WithCacheDir(dir string) Option
  • func WithConcurrency(n int) Option
  • func WithHTTPClient(client *http.Client) Option
  • func WithLogger(l *slog.Logger) Option
  • func WithProgress(fn func(Progress)) Option
  • func WithRateLimit(weightPerSecond float64) Option
• type Progress
• type PruneOptions
• type PruneResult
• type Request
  • func (r Request) Validate() error
• type Source
  • func (s Source) String() string

Examples

• Package
• Package (Errors)
• Availability.MonthlyGaps
• Closes
• Interval.Duration
• Interval.HasDailyArchives
• Loader.Available
• Loader.FetchAll
• Loader.Stream
• Loader.VerifyCache
• NewLoader
• NormalizeSymbol
• ParseInterval
• Request.Validate
• WithProgress
• WriteParquet

Constants

const (
	// CodecVersion identifies the decoding rules implemented in this file. It
	// is stamped into every derived cache file and compared on read.
	//
	// It exists because "same ZIP implies same output" is only true while the
	// *conversion* is unchanged. Fix a parsing bug — the millisecond handling
	// above is the obvious candidate — and every cache entry built by the old
	// code is now wrong while its source archive is byte-for-byte identical.
	// No checksum can detect that, because nothing about the source changed.
	//
	// So: bump this constant in the same commit as any change to what this
	// file produces from given bytes, and every stale cache entry rebuilds
	// itself on next read, offline. Leaving it alone after such a change means
	// the corrected parser never reaches data that is already cached.
	//
	// Being a compile-time constant is load-bearing. Two runs of one binary
	// cannot disagree about it, so a cache entry either matches the running
	// code or does not, with no third state to reason about.
	CodecVersion = 1
)

This file is the codec: the one place where the format Binance publishes meets the types this package defines. Everything upstream of it deals in URLs and bytes and knows nothing about candles; everything downstream deals in Kline values and knows nothing about CSV.

What is actually in an archive

A bulk archive is a ZIP holding exactly one CSV member, named after the archive itself — BTCUSDT-1h-2024-01-15.zip contains BTCUSDT-1h-2024-01-15.csv. Spot files carry no header row, and every data row has twelve fields:

1705276800000,41732.35000000,42353.94000000,41718.05000000,42279.75000000,
2433.52283000,1705280399999,102456168.22559890,66725,1348.93805000,
56779695.64663060,0

Reading that into a struct is the easy part. Three things are not, and each is a bug this library exists to not repeat:

• Whether a header row is present is *sniffed*, never assumed. Spot has none and futures has one, and a wrong guess either eats a real candle or parses the word "open_time" as a price.
• The timestamp unit is decided *per row*. Binance switched these files from milliseconds to microseconds at 2025-01-01T00:00Z — verified here against the real archives for 2024-12-31 and 2025-01-01, which are the last and first days of each. The ported implementation sniffed the unit from the final row of a file and applied it to all of them.
• Every number is parsed as an exact decimal. This is the hot path that motivates the whole two-tier cache: eight udecimal.Decimal fields times 44,640 rows in a month of one-minute candles.

Rows are yielded, not returned

The entry points below hand back an iter.Seq2 — a range-over-function iterator, Go's equivalent of a Python generator — rather than a []Kline. That is a memory decision. A month of 1s candles is about 2.6 million rows, and a Kline is 312 bytes, so materialising one archive would cost 810 MB. Stage 5 writes the cache row by row from this iterator and never holds an archive whole; an unexported helper collects one into a slice for the callers that genuinely want one.

const DevVersion = "(devel)"

DevVersion is reported by Version when the binary carries no version at all. That is what `go run` produces, and what a `go test` binary carries, so it is the string the test suite sees.

It is *not* what a plain `go build` produces inside this repository. Since Go 1.24 the toolchain reads the version control system and stamps what it finds, so a build here carries a real version even with no release involved — see Version for the four cases, measured. DevVersion comes back from a build only when version control is unavailable or switched off with -buildvcs=false.

The string itself is Go's own convention, so it matches what `go version -m` prints for the same binary.

Variables

var (
	// ErrInvalidRequest reports that a request was rejected before any I/O
	// happened, because something about the request itself is wrong: an
	// unknown interval, a malformed symbol, a start after its end, an interval
	// Binance does not publish at the granularity being asked for.
	//
	// This error is always the caller's to fix, and it is always cheap: no
	// network round trip is spent discovering it.
	ErrInvalidRequest = errors.New("invalid request")

	// ErrNotAvailable reports that Binance does not have the requested data —
	// an archive URL that answers 404, a symbol that was not yet listed on the
	// date asked for, or a day too recent to have been published.
	//
	// This is a fact about the world, not a failure of the program, and it is
	// routinely the correct answer. The download layer converts a 404 into
	// this error so that "no data here" arrives as a value the caller can
	// branch on.
	//
	// The Python implementation this library replaces returned a None
	// DataFrame for a 404 and relied on every call site remembering to check.
	// Several did not. Here, ignoring the error is a visible act.
	ErrNotAvailable = errors.New("data not available")

	// ErrChecksum reports that a downloaded or cached archive did not match
	// the SHA-256 in its .CHECKSUM sidecar. Either the transfer was corrupted
	// or the cached file was damaged on disk; in both cases the bytes are not
	// what Binance published and must not be parsed.
	//
	// The Python implementation wrote checksums to disk and never verified
	// them on read, which makes the sidecar decorative. Here a mismatch is
	// load-bearing: it discards the file and re-downloads.
	ErrChecksum = errors.New("checksum mismatch")

	// ErrCorruptArchive reports that bytes which passed transport and checksum
	// checks still could not be understood: a ZIP whose central directory will
	// not open, an archive holding no CSV member, a row with the wrong number
	// of fields, a price that will not parse as a decimal.
	//
	// It is kept distinct from ErrChecksum because the two call for different
	// responses. A checksum failure is transient and worth retrying; a corrupt
	// archive that verified correctly means Binance published something this
	// parser does not understand, and retrying will produce the same bytes.
	ErrCorruptArchive = errors.New("corrupt archive")

	// ErrRateLimited reports that Binance asked us to slow down — HTTP 429, or
	// 418 after repeated 429s. The REST tail of a request is the only place
	// this can occur; the bulk archives are static files and are not limited.
	//
	// It is a distinct sentinel because it is the one failure where the right
	// response is to wait rather than to give up or to try elsewhere. A 418
	// carries [ErrIPBanned] as well, for the case where waiting is not enough.
	ErrRateLimited = errors.New("rate limited")

	// ErrIPBanned reports an HTTP 418: Binance has barred this IP address
	// rather than merely asked it to slow down. It is the escalation applied
	// to a client that keeps ignoring 429s, and it lasts from two minutes to
	// three days, lengthening with repeat offences.
	//
	// Every error carrying it also carries ErrRateLimited, so a caller asking
	// only "should I slow down?" needs to know nothing about it. The reason it
	// exists as well is that the right response is different in kind: there is
	// no backoff short enough to ride out a ban, and retrying is what earns the
	// next, longer one. A pipeline that can stop should stop.
	//
	// The ban is on the address, not the process or the API key. One earned by
	// a history download also locks out anything else on the same host — a live
	// trading bot, most expensively — which is why internal/vision paces this
	// endpoint preventatively rather than reacting once this arrives.
	ErrIPBanned = errors.New("ip banned")
)

This file holds every sentinel error the package can return. Keeping them in one place is a deliberate choice: the set of things that can go wrong is part of the public API, and a reader should be able to learn it from a single screen rather than by grepping for errors.New.

What an error is in Go

There are no exceptions. `error` is an ordinary interface with one method:

type error interface { Error() string }

Any type implementing it is an error, and errors travel as ordinary return values. A function that can fail returns one as its last result, and the caller checks it. That is the whole mechanism.

What a sentinel is

A sentinel is a package-level error variable that stands for one specific condition, so callers can ask "was it *this* problem?" rather than matching on message text. errors.New returns a pointer to a newly allocated struct, which matters more than it looks: two calls to errors.New("boom") produce two distinct, non-equal errors. Identity is the thing being compared, never the string — so the messages below can be reworded without breaking any caller.

Wrapping, and why %w

A sentinel alone is too coarse: "invalid request" does not say which field. So the site that detects the problem adds context by wrapping:

return fmt.Errorf("interval %q: %w", s, ErrInvalidRequest)

The %w verb (as opposed to %v) records the wrapped error inside the new one, building a chain that errors.Is can walk. The result prints as `interval "2h30m": invalid request` and still answers true to errors.Is(err, ErrInvalidRequest).

Always errors.Is, never ==

Comparing with == tests only the outermost error, so it silently returns false the moment anyone wraps. errors.Is walks the whole chain, so it keeps working. The errorlint linter (see .golangci.yml) fails the build on ==, which is how this rule stays true as the codebase grows.

if errors.Is(err, ErrNotAvailable) { ... }   // correct
if err == ErrNotAvailable { ... }            // broken by any wrapping

A `var` block groups related declarations the way an import block groups imports. It is one declaration, not six, and reads as a single unit.

Functions

func Closes

func Closes(klines []Kline) []float64

Closes returns the close price of every candle as a float64 slice. It is the column most indicators want. See Opens for the precision caveat.

Example

ExampleCloses shows extracting a price column for an indicator library, and the precision that is given up by doing so.

package main

import (
	"fmt"

	"github.com/quagmt/udecimal"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	klines := []binancedata.Kline{
		{Close: udecimal.MustParse("42283.58000000")},
		{Close: udecimal.MustParse("42580.00000000")},
	}

	fmt.Println(binancedata.Closes(klines))

	// The column helpers return []float64 because that is what every Go
	// technical-indicator package takes, and the conversion is lossy by
	// definition. The exact value is still on the Kline, so arithmetic that
	// has to balance — a portfolio total, a fee calculation — should read the
	// udecimal.Decimal field rather than the column.
	fmt.Println(klines[0].Close)

	// Note which column has no helper: QuoteVolume. It is the field that
	// reaches twenty significant digits, which is what ruled out float64 for
	// the whole struct in the first place, so there is no float64 slice of it
	// to hand out.

}

Output:
[42283.58 42580]
42283.58

func Highs

func Highs(klines []Kline) []float64

Highs returns the high price of every candle as a float64 slice. See Opens for the precision caveat.

func Lows

func Lows(klines []Kline) []float64

Lows returns the low price of every candle as a float64 slice. See Opens for the precision caveat.

func NormalizeSymbol

func NormalizeSymbol(s string) (string, error)

NormalizeSymbol converts the ways a trading pair is commonly written into the single form Binance uses in URLs and file names. All three of "BTC/USDT", "BTC-USDT" and "btcusdt" become "BTCUSDT".

Surrounding whitespace is trimmed, the separators "/" and "-" are removed, and ASCII letters are upper-cased. Anything that survives that and is still not an ASCII letter or digit is an error, as is a result shorter than 3 characters or longer than 20. The returned error wraps ErrInvalidRequest.

Normalising early matters for more than tidiness. The symbol becomes part of a URL, part of a cache path and part of a cache key, so two spellings of one pair that reach those layers unnormalised produce two cache entries, two downloads, and a "why is my cache twice the size" question much later.

An underscore is *not* stripped, though it is a plausible separator. Binance delivery-futures contracts are named BTCUSDT_240329, so stripping it would corrupt a real symbol the moment futures support arrives. It is rejected today because spot symbols never contain one; that rejection is the single line to relax at the futures extension point.

Example

ExampleNormalizeSymbol shows the three spellings that mean one pair.

package main

import (
	"errors"
	"fmt"
	"log"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	// A human writes BTC/USDT, a config file often has BTC-USDT, and Binance's
	// own URLs use BTCUSDT. All three are accepted so that a caller never has
	// to reformat, and all three normalise to the form the API and the archive
	// paths expect.
	for _, s := range []string{"BTC/USDT", "btc-usdt", "BTCUSDT"} {
		norm, err := binancedata.NormalizeSymbol(s)
		if err != nil {
			log.Fatal(err)
		}

		fmt.Println(norm)
	}

	// Anything outside ASCII letters and digits is rejected rather than
	// stripped, because a symbol quietly rewritten into a different valid
	// symbol is worse than one that fails.
	_, err := binancedata.NormalizeSymbol("BTC USDT")
	fmt.Println(errors.Is(err, binancedata.ErrInvalidRequest))

}

Output:
BTCUSDT
BTCUSDT
BTCUSDT
true

func OpenTimes

func OpenTimes(klines []Kline) []time.Time

OpenTimes returns the open time of every candle, which is what a plotting or resampling routine needs alongside a price column.

func Opens

func Opens(klines []Kline) []float64

Opens returns the open price of every candle as a float64 slice.

The conversion is inexact: float64 cannot represent every decimal Binance publishes. It is the right trade for feeding a technical-indicator library, since every Go indicator package takes []float64 — but do the arithmetic that has to balance on the udecimal.Decimal fields instead.

func Version

func Version() string

Version reports the version of this module that was linked into the running binary, for example "v0.3.1", or DevVersion when the binary carries none.

There is no version constant to keep in sync anywhere in this repository, and no build script injecting one with -ldflags. The Go toolchain stamps the module version into every binary at link time and this function reads it back out, so a release is made by pushing a git tag and nothing in the source needs editing. A second source of truth is the thing this design exists to avoid: a constant and a tag that disagree is a bug nobody notices until they are debugging the wrong version.

What the toolchain actually stamps

Worth spelling out, because it decides whether a release pipeline needs to inject anything. Measured with `go version -m` on Go 1.26:

go run ./cmd/bmd                     (devel)
go build with -buildvcs=false        (devel)
go build, untagged commit            v0.0.0-20260821111323-f4255484548a
go build, clean tree at a tag        v0.1.0
go build, dirty tree at a tag        v0.1.0+dirty

The last two are why goreleaser is configured with no version ldflags: it builds from a checkout at the tag, so the tag arrives on its own. The +dirty suffix is a bonus — a binary built from uncommitted changes says so rather than impersonating the release.

func Volumes

func Volumes(klines []Kline) []float64

Volumes returns the base-asset volume of every candle as a float64 slice. See Opens for the precision caveat, which bites hardest on volume columns.

func WriteParquet

func WriteParquet(ctx context.Context, w io.Writer, seq iter.Seq2[Kline, error]) (int, error)

WriteParquet writes candles to w as a parquet file, and returns how many it wrote.

It is the export half of the format the cache stores its second tier in: same schema, same column types, same writer settings, so a file written here and a file written by the cache differ only in their footer metadata. Reach for it when you want candles on disk in a form a query engine can read — DuckDB, Polars, pandas, Spark — without going through CSV and its float64s.

f, err := os.Create("btcusdt-1h-2024.parquet")
// ...
n, err := binancedata.WriteParquet(ctx, f, loader.Stream(ctx, req))

Why it takes an iterator

Because Loader.Stream produces one, and the pair of them is what lets a range larger than memory reach a file. A []Kline parameter would have forced the whole range to exist at once, which is the thing Stream is for avoiding — five years of one-minute candles is about 820 MB of Kline. A caller who *does* have a slice can pass slices.All(klines) with an error-free adapter, which is a cheap wrapper; the reverse is not cheap at all.

What it does not write

No source stamp. The cache binds each tier-2 file to the archive it was built from with a SHA-256 in the footer, because a derived file that cannot prove its provenance is one the read path would have to accept on trust. An export has no single archive behind it — a year's worth of candles comes from twelve archives and possibly the REST API — so there is nothing honest to put there, and inventing a value would make a file that reads as a cache entry without being one. The codec version and the row count are written, because both are true of any file this function produces.

Reading one back

With whatever already reads parquet, which is the point of writing one. There is deliberately no ReadParquet in this package, and the asymmetry is the answer rather than an omission: a Go program that wants candles has Loader.Fetch, which reads the same format out of the cache in about 6 ms per symbol-month and hands back Kline values — exporting and re-reading in the same process would be a slower way to do that. Exports exist for the tools that are not this program.

The schema is fixed and documented in docs/caching.md, so a reader needs nothing from here: eleven columns in Kline's own order, open_time and close_time as TIMESTAMP(MICROS), the eight money columns as DECIMAL(38,8) in a FIXED_LEN_BYTE_ARRAY(16), and trades as an INT64. The footer carries bmd.codec.version and bmd.rows.

Errors

Nothing is written to w after an error, but what was already written stays written: parquet puts its footer last, so an interrupted export leaves bytes no reader will accept rather than a short file that looks complete. If that matters, write to a temporary file and rename it once this returns nil — which is what the cache does, and for exactly this reason.

Example

ExampleWriteParquet shows exporting a range for a query engine to read.

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	f, err := os.Create("btcusdt-1h-2024.parquet")
	if err != nil {
		log.Fatal(err)
	}

	// The iterator from Stream feeds straight in, which is what lets a range
	// larger than memory reach a file. Prices land as DECIMAL(38,8) and times
	// as TIMESTAMP(MICROS), so DuckDB, Polars or pandas read the exact values
	// rather than float64 approximations of them.
	n, err := binancedata.WriteParquet(ctx, f, loader.Stream(ctx, binancedata.Request{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1h,
		Market:   binancedata.MarketSpot,
		Start:    time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC),
		End:      time.Date(2024, 12, 31, 23, 0, 0, 0, time.UTC),
	}))
	if err != nil {
		// Parquet writes its footer last, so an interrupted export leaves
		// bytes no reader will accept rather than a short file that looks
		// complete. Write to a temporary file and rename on success if that
		// distinction matters.
		log.Fatal(err)
	}

	// Close explicitly and check the error, rather than only deferring it.
	// WriteParquet has already written the footer by the time it returns — it
	// closes the parquet writer itself — but the bytes may still be sitting in
	// the operating system's buffers, and Close is where a delayed write
	// failure surfaces: a full disk, a network filesystem that gave up. A
	// deferred close whose error is discarded reports success for a file that
	// was never fully written.
	if err := f.Close(); err != nil {
		log.Fatal(err)
	}

	fmt.Println(n, "candles written")
}

Types

type Availability

type Availability struct {
	// Symbol, Interval and Market echo the query, with the symbol normalised.
	// They are here so that a result can be passed around, logged or printed
	// without its question having to travel beside it.
	Symbol   string
	Interval Interval
	Market   Market

	// Monthly and Daily hold the start instant of every archive the bucket
	// listed, ascending: midnight UTC on the 1st for a month, midnight UTC for
	// a day. Both are nil for a symbol that never traded — which is a fact
	// about Binance, not an error, and is why [Loader.Available] returns it
	// with a nil error.
	//
	// Daily is always nil for the three intervals Binance publishes monthly
	// only: 3d, 1w and 1mo.
	Monthly []time.Time
	Daily   []time.Time

	// ArchivesThrough is the first instant no archive covers. Everything at or
	// after it has to come from the REST API, and it is the frontier the
	// planner uses. The zero time means there are no archives at all.
	ArchivesThrough time.Time
}

Availability is what Binance publishes for one symbol and interval: which archives exist, and where the archives stop and the REST API takes over.

It answers the question a caller cannot answer from a calendar — "how far back does this pair go, and is anything missing in the middle?" — and it answers it by asking the bucket rather than by inferring. Binance is not contractually bound to publish on any schedule, and it demonstrably has holes: BTCUSDT-1mo-2024-03.zip does not exist while 2024-02 and 2024-04 both do. No date arithmetic predicts that.

func (Availability) DailyGaps

func (a Availability) DailyGaps() []time.Time

DailyGaps returns the days missing between the first and last daily archive. See Availability.MonthlyGaps.

func (Availability) MonthlyGaps

func (a Availability) MonthlyGaps() []time.Time

MonthlyGaps returns the months missing between the first and last monthly archive. DailyGaps does the same for days.

Only the interior counts. Periods after the last archive are not gaps — they are the part of history that has not been published yet, which is what Availability.ArchivesThrough is for — and there is nothing before the first archive to have a hole in.

A non-empty result is the case that breaks calendar-based implementations, and it is why this library lists the bucket at all.

Example

ExampleAvailability_MonthlyGaps shows the case that breaks every calendar-based implementation.

package main

import (
	"fmt"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	month := func(y int, m time.Month) time.Time {
		return time.Date(y, m, 1, 0, 0, 0, 0, time.UTC)
	}

	// These are the real monthly archives Binance publishes for BTCUSDT at 1mo
	// around early 2024, with the real hole in them: 2024-02 and 2024-04 exist,
	// 2024-03 does not. Loader.Available returns this by listing the bucket;
	// it is written out by hand here so the example needs no network.
	a := binancedata.Availability{
		Symbol:   "BTCUSDT",
		Interval: binancedata.Interval1mo,
		Market:   binancedata.MarketSpot,
		Monthly: []time.Time{
			month(2024, time.January),
			month(2024, time.February),
			month(2024, time.April),
			month(2024, time.May),
		},
		ArchivesThrough: month(2024, time.June),
	}

	for _, gap := range a.MonthlyGaps() {
		fmt.Println(gap.Format("2006-01"))
	}

	// Only interior holes count. May is the last archive and June is where the
	// REST API takes over, so nothing after May is a gap — it is simply
	// unpublished, which is what ArchivesThrough says.
	fmt.Println("archives run out at", a.ArchivesThrough.Format("2006-01"))

}

Output:
2024-03
archives run out at 2024-06

type AvailabilityQuery

type AvailabilityQuery struct {
	// Symbol is the trading pair, in any spelling [NormalizeSymbol] accepts.
	Symbol string

	// Interval is the candle period. Required; the zero value is invalid.
	Interval Interval

	// Market selects the Binance market. Required; the zero value is invalid.
	Market Market

	// Since bounds the answer, and bounds its cost. The bucket listing is
	// seeked with a marker built from it, so asking about 2024 onwards is one
	// round trip where asking about everything is seven for a pair that has
	// traded since 2017. Optional; the zero value means the whole history.
	//
	// It is not a filter applied afterwards. Archives before it are not
	// listed, so they are absent from the result rather than excluded from it,
	// and [Availability.ArchivesThrough] is the only field a Since cannot
	// affect — the frontier is at the far end.
	Since time.Time
}

AvailabilityQuery names what to ask the bucket about. It is the input to Loader.Available.

It is a struct rather than four parameters for the same reason Request is: three of the fields are required and one is not, and a struct says which is which at the call site. It is a *separate* struct from Request because Request carries a Start and an End that this question has no use for, and a type whose fields are silently ignored is the kind of API that teaches callers to stop trusting the ones that are not.

type CacheEntry

type CacheEntry struct {
	// Path is the archive's absolute path on disk.
	Path string

	// Sidecar is the absolute path of the .CHECKSUM file that names this
	// archive's published hash. It is always set, including when reading it is
	// what failed, because a caller acting on a failed entry needs to name both
	// halves of it.
	//
	// It is a field rather than something a caller derives, and the reason is
	// worth stating: deriving it means knowing that the suffix is ".CHECKSUM"
	// and that it is appended to the whole file name rather than replacing the
	// extension. That is Binance's naming rule, not this library's, and a
	// caller who hardcodes it has taken on a rule it cannot see change. The
	// cost of the rule moving is silent — a delete that removes the archive and
	// leaves an orphan — which is the kind of coupling a field cheaply removes.
	Sidecar string

	// Size is the archive's size in bytes, which is what makes a progress
	// display possible: hashing is proportional to it.
	Size int64

	// Err is nil when the archive's bytes hash to the value in its .CHECKSUM
	// sidecar. Otherwise it says why not, and the three answers call for
	// different responses:
	//
	//   - wrapping [ErrChecksum]: the bytes on disk are not what Binance
	//     published. Delete the file; the next fetch downloads it again.
	//   - wrapping [fs.ErrNotExist]: half an entry. The cache writes the
	//     archive first and the sidecar second, so a crash between the two
	//     leaves exactly this. It is not corruption, and the read path already
	//     treats it as a cache miss — but what is left cannot be verified or
	//     used, so deleting it is safe and reclaims the space.
	//   - anything else: an I/O failure reading one of the two files, or a
	//     sidecar whose contents will not parse. This is a fact about the disk
	//     rather than about the data, and the archive may be perfectly good.
	//     Do not delete on this one; report it and let a person look.
	//
	// Compare with [errors.Is], never with ==. The first two cases are
	// deliberately distinguishable that way, because "delete it" and "leave it
	// alone" is the decision every caller of [Loader.VerifyCache] has to make.
	Err error
}

CacheEntry is one cached archive and what verifying it found.

type CacheUsage

type CacheUsage struct {
	// Root is the directory these numbers describe, absolute.
	Root string

	// Archives and ArchiveCount are tier 1 — the .zip files exactly as Binance
	// served them.
	//
	// This is the smaller of the two tiers, which is worth knowing before
	// deciding what pruning is worth. Measured on BTCUSDT 1m for 2024-01: the
	// archive is 2,169,570 bytes and the parquet built from it is 3,226,820, so
	// tier 1 is about 40% of the entry. Tier 2 is larger on purpose — snappy
	// where the zip uses deflate, and fixed-width DECIMAL(38,8) where the CSV
	// uses text — because it is the tier that is read, and docs/caching.md
	// costs the whole trade at roughly 2× the archive alone.
	Archives     int64
	ArchiveCount int

	// Sidecars and SidecarCount are the .CHECKSUM files. Around ninety bytes
	// each — 64 hex digits, two spaces and the archive's own file name, so the
	// size moves with the length of that name and nothing here depends on a
	// fixed figure. Either way they will never matter to a disk-space decision,
	// and they are counted
	// separately anyway because of what they mean: the hash in a sidecar is what
	// the parquet beside it is validated against, so a sidecar is the one part
	// of tier 1 that pruning must leave behind. Deleting them would strand
	// tier 2 exactly as deleting tier 2 would.
	Sidecars     int64
	SidecarCount int

	// Parquet and ParquetCount are tier 2 — the derived files that reads
	// actually hit.
	Parquet      int64
	ParquetCount int

	// Other and OtherCount are every file under the root that is none of the
	// three. Both are zero in a healthy cache, and they are reported rather
	// than quietly folded into a total because a non-zero value has exactly one
	// ordinary cause worth knowing about: every cache write goes to a temporary
	// file in its destination directory, so a process killed mid-write leaves
	// one behind and nothing ever collects it.
	Other      int64
	OtherCount int

	// Prunable and PrunableCount are the archives a prune would delete — those
	// whose parquet can already serve reads without them. See
	// [Loader.PruneArchives] for what that means and what it costs.
	//
	// This is a subset of Archives and is deliberately not subtracted from it.
	Prunable      int64
	PrunableCount int
}

CacheUsage is what a cache directory holds, counted in files and bytes.

The four size categories are disjoint and cover every file under the root, so they sum to CacheUsage.Total. Prunable is not one of them: it is a subset of Archives, counted again because it is the number a caller deciding whether to prune actually wants.

func (CacheUsage) Total

func (u CacheUsage) Total() int64

Total is every byte under the cache root.

It sums the sizes of the files themselves, so expect it to read slightly under du(1), which counts whole filesystem blocks and the directories too.

type EvictOptions

type EvictOptions struct {
	// Symbols limits the eviction to these trading pairs, in any spelling
	// [NormalizeSymbol] accepts. Empty means every symbol.
	Symbols []string

	// Intervals limits it to these candle periods. Empty means every interval.
	Intervals []Interval

	// Before limits it to entries that end at or before this instant —
	// entries covering *only* instants earlier than it.
	//
	// The bound is on the data, not on the file: it is read from the archive's
	// own name, so it means the same thing whenever the entry was downloaded.
	// It is also exclusive at the whole-entry level rather than trimming, since
	// an entry is the unit the cache stores: January's monthly archive survives
	// a Before of 2024-01-15, because half of it is still wanted and there is
	// no way to keep half a file. Set Before to 2024-02-01 to remove it.
	//
	// Must be UTC when set. The zero value means every period.
	Before time.Time

	// All evicts the entire cache. It is the only way to run with no filter,
	// and setting it alongside one is an error rather than a redundancy — a
	// call that says both "everything" and "only these" has two readings and
	// neither is safe to guess.
	All bool

	// DryRun reaches every verdict and deletes nothing. Each [EvictResult]
	// comes back with the files it would have removed and Removed false.
	DryRun bool
}

EvictOptions selects which cache entries Loader.EvictCache removes.

Why every field is a filter and none is a policy

Because the two automatic policies a cache usually gets are both unreliable here, and it is worth saying which and why rather than leaving the absence to look like an oversight.

*Expire by age* would have to read a file's modification time, and that records when an entry was **downloaded**, not when it was last used. A symbol-month a backtest reads on every run expires on schedule while one fetched yesterday and never opened again survives — the wrong axis, applied confidently.

*Evict least-recently-used under a size cap* needs a recency signal the filesystem does not reliably give. Access times are off or coarse on most Linux configurations (`noatime`, or `relatime`'s once-a-day update), so the library would have to record reads itself: a write on every cache hit, against a read path whose whole claim is that a hit opens the sidecar and a parquet footer and touches nothing else. Paying for that on every read to answer a question asked once a month is the wrong trade.

What is left is the signal the caller actually has, and it is a good one: "my window moved on, drop 2019", "I am done with these symbols". So this is a selection rather than a rule, and the deleting happens when it is asked for.

At least one selector is required

A zero EvictOptions is an error rather than "everything", because the zero value of a struct is what a caller gets by forgetting to fill one in, and the cost of that mistake here is the whole cache. Deleting everything is a legitimate request and has its own spelling: All, which may not be combined with a filter.

type EvictResult

type EvictResult struct {
	// Name is the entry's archive name without its extension, e.g.
	// "BTCUSDT-1h-2024-01". It is what to print: the date in it is formatted
	// the way the entry's own granularity formats it, so a monthly entry reads
	// as 2024-01 and a daily one as 2024-01-15 without the caller deciding.
	Name string

	// Symbol and Interval are the entry's, taken from the cache tree it sits
	// in rather than parsed out of Name.
	Symbol   string
	Interval Interval

	// Period is the first instant the entry covers, UTC. Whether it spans a
	// month or a day is visible in Name.
	Period time.Time

	// Files are the entry's files that were on disk, absolute, in the order
	// archive, sidecar, parquet — skipping any that was already gone.
	Files []string

	// Size is their total in bytes: what evicting this entry reclaims, or
	// would have.
	Size int64

	// Removed reports whether this call deleted them. It is false in a dry
	// run, and false when Err is set.
	//
	// A partial failure counts as not removed: if one of the three files could
	// not be deleted, Err says so and Removed stays false even though the
	// others are gone. The entry is unusable either way — a read needs all
	// three or a rebuild — and reporting it as removed would overstate what
	// was reclaimed.
	Removed bool

	// Err is set when a file should have gone and would not.
	Err error
}

EvictResult is one cache entry an eviction considered, and what it decided.

An entry is the unit the cache stores and therefore the unit this removes: the archive, its .CHECKSUM sidecar and the parquet built from it, however many of the three are still on disk. A previously pruned entry has two of them and is evicted exactly like any other.

type Interval

type Interval uint8

Interval is a kline (candlestick) aggregation period: 1m, 1h, 1d and so on.

Why this is a type and not a string

The obvious representation is a plain string, and it is the wrong one. A string parameter accepts "1hour", "60m", "" and "DROP TABLE" equally happily, and the mistake surfaces as a 404 several layers away from the typo. Defining a named type moves that check to a single place — ParseInterval — and lets the compiler carry the guarantee everywhere afterwards. A function taking an Interval cannot be handed an arbitrary string by accident.

This is the single most common Go idiom worth importing into your habits from Python: where you would reach for str or an Enum, define a named type over a small integer, hang methods on it, and let the type do the arguing.

The two spellings

The same interval is spelled differently by the two Binance endpoints this library uses, and exactly one interval disagrees:

bulk archives (data.binance.vision)   monthly candles are "1mo"
REST API (data-api.binance.vision)    monthly candles are "1M"

Worse, the REST spelling is case-sensitive in a hostile way: "1m" is one minute and "1M" is one month, so a stray strings.ToUpper turns a minute into a month and returns plausible-looking wrong data. Rather than let each endpoint's code carry its own string, an Interval knows both spellings — Interval.String gives the archive one and Interval.RESTParam the REST one — so the two paths cannot drift apart.

Not every interval exists everywhere

Binance does not publish every interval at both archive granularities: 3d, 1w and 1mo exist as monthly archives only, since their candles are longer than a day. Ask for the wrong combination and you get a 404 that looks exactly like "this symbol did not trade yet". Interval.HasDailyArchives and Interval.HasMonthlyArchives answer this before any request is made.

The Python implementation this library replaces declares these same tables and then never consults them — and its monthly table is wrong: it omits 1s, which Binance does publish monthly. Verified against the live archives on 2026-08-18; see the note on the table below.

const (
	Interval1s  Interval = iota + 1 // 1 second
	Interval1m                      // 1 minute
	Interval3m                      // 3 minutes
	Interval5m                      // 5 minutes
	Interval15m                     // 15 minutes
	Interval30m                     // 30 minutes
	Interval1h                      // 1 hour
	Interval2h                      // 2 hours
	Interval4h                      // 4 hours
	Interval6h                      // 6 hours
	Interval8h                      // 8 hours
	Interval12h                     // 12 hours
	Interval1d                      // 1 day
	Interval3d                      // 3 days — monthly archives only
	Interval1w                      // 1 week — monthly archives only
	Interval1mo                     // 1 calendar month — monthly archives only
)

The intervals Binance publishes klines for.

Reading the iota

iota is Go's constant generator. Inside a const block it counts from 0, incrementing once per ConstSpec line, and a line that omits its expression repeats the previous one — so writing the expression once on the first line defines the whole ladder.

The `+ 1` is load-bearing. It leaves 0 unassigned, which makes the zero value of Interval — what you get from `var iv Interval` or an unset struct field — an invalid interval rather than a silently plausible one. Go has no constructors and cannot stop a caller from writing binancedata.Request{}, so "the zero value is detectably wrong" is the only defence available for a field a caller must actually choose. Market is built the same way, for the same reason, as is the unexported dataType beside it.

The names are terse on purpose: Interval1h reads at a call site the way the documentation reads, and the type name already supplies the noun.

func Intervals

func Intervals() []Interval

Intervals returns every valid interval, ordered from shortest to longest.

The slice is freshly built on each call, which is deliberate. Returning a package-level slice would hand callers a window onto this package's own memory — slices are views over a shared backing array, so a caller writing to element 0 would corrupt the library for everyone in the process. Copying a sixteen-element slice is cheaper than that class of bug.

func ParseInterval

func ParseInterval(s string) (Interval, error)

ParseInterval converts a spelling of an interval into an Interval. It accepts both the archive form and the REST form, so "1mo" and "1M" both return Interval1mo.

Matching is exact and case-sensitive, and that is a correctness requirement rather than strictness for its own sake: Binance uses "1m" for one minute and "1M" for one month. Case-folding the input would quietly turn a request for minute candles into a request for monthly ones — the same number of rows never comes back, but the data looks superficially fine.

The returned error wraps ErrInvalidRequest, so callers test it with errors.Is rather than by inspecting the message.

Example

ExampleParseInterval shows the case-sensitivity that matters most.

package main

import (
	"errors"
	"fmt"
	"log"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	// Binance spells one month two ways, depending on which endpoint is
	// asking: "1mo" in the archive paths, "1M" in the REST API. Both parse to
	// the same value, so a caller never has to know which one they were
	// handed.
	archive, err := binancedata.ParseInterval("1mo")
	if err != nil {
		log.Fatal(err)
	}

	rest, err := binancedata.ParseInterval("1M")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(archive == rest, archive)

	// The other half of that: "1m" is one *minute*. Matching is exact and
	// case-sensitive precisely so this pair cannot be confused, because the
	// confusion is silent — monthly candles for a request that wanted minutes
	// still look like candles.
	minute, err := binancedata.ParseInterval("1m")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(minute)

	// An unknown spelling wraps ErrInvalidRequest, so it is tested with
	// errors.Is rather than by reading the message.
	_, err = binancedata.ParseInterval("1 hour")
	fmt.Println(errors.Is(err, binancedata.ErrInvalidRequest))

}

Output:
true 1mo
1m
true

func (Interval) Duration

func (i Interval) Duration() (time.Duration, bool)

Duration returns the wall-clock length of one candle at this interval, and whether that length is fixed at all.

The second result is false for Interval1mo, whose candles are 28, 29, 30 or 31 days long depending on where in the calendar they fall, and for an invalid interval. Returning (0, false) rather than a plausible-looking 30*24h is the point: a caller computing "how many candles should this range contain?" needs to be told that arithmetic does not apply here, not handed an approximation that is wrong eleven months a year.

Multiple return values are how Go says "and also"; there is no tuple type and no out-parameter. The (value, ok) shape specifically mirrors what map reads and type assertions already do, so it reads as familiar rather than novel.

Example

ExampleInterval_Duration shows the interval whose candles have no fixed length, and why that is a second return value rather than an approximation.

package main

import (
	"fmt"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	d, ok := binancedata.Interval1h.Duration()
	fmt.Println(d, ok)

	// A calendar month is 28, 29, 30 or 31 days depending on where it falls,
	// so there is no duration to return. The false is the useful part: a
	// caller computing "how many candles should this range hold?" is told the
	// arithmetic does not apply, instead of being handed 30*24h and being
	// wrong eleven months a year.
	d, ok = binancedata.Interval1mo.Duration()
	fmt.Println(d, ok)

}

Output:
1h0m0s true
0s false

func (Interval) HasDailyArchives

func (i Interval) HasDailyArchives() bool

HasDailyArchives reports whether Binance publishes daily ZIP archives for this interval. It is false for Interval3d, Interval1w and Interval1mo, whose candles are longer than a day.

Example

ExampleInterval_HasDailyArchives shows how to tell which archives exist for an interval before asking for any.

package main

import (
	"fmt"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	// Binance publishes daily archives for most intervals but not for the
	// three coarsest: a 1w candle spans more than a day, so a daily file could
	// not hold a whole one. The planner already knows this; the method is here
	// for a caller building their own UI over the same rules.
	for _, i := range []binancedata.Interval{
		binancedata.Interval1h,
		binancedata.Interval1d,
		binancedata.Interval1w,
		binancedata.Interval1mo,
	} {
		fmt.Printf("%-4s monthly=%-5t daily=%t\n", i, i.HasMonthlyArchives(), i.HasDailyArchives())
	}

}

Output:
1h   monthly=true  daily=true
1d   monthly=true  daily=true
1w   monthly=true  daily=false
1mo  monthly=true  daily=false

func (Interval) HasMonthlyArchives

func (i Interval) HasMonthlyArchives() bool

HasMonthlyArchives reports whether Binance publishes monthly ZIP archives for this interval. It is true for every interval — including Interval1s, whose monthly archives are real but large, around 93 MB compressed for BTCUSDT.

func (Interval) IsValid

func (i Interval) IsValid() bool

IsValid reports whether i is one of the intervals Binance publishes.

The zero value is not, which is what makes an unset struct field detectable.

func (Interval) MarshalText

func (i Interval) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler, writing the archive spelling.

The standard library looks for this interface in several places at once: encoding/json uses it for map keys and struct fields, and flag.TextVar builds a command-line flag out of any type that implements it and its unmarshalling counterpart. Implementing two small methods therefore buys JSON round-tripping and CLI parsing without either package knowing this type exists.

func (Interval) RESTParam

func (i Interval) RESTParam() string

RESTParam returns the spelling the REST API expects in its `interval` query parameter, which differs from Interval.String only for Interval1mo: "1M" rather than "1mo".

It returns the empty string for an invalid interval. Callers building a request should reject the interval before they get here — an empty parameter would be answered with a 400, which is a far less clear diagnosis than ErrInvalidRequest raised at the boundary.

func (Interval) String

func (i Interval) String() string

String returns the archive spelling of the interval — the form that appears in data.binance.vision paths, where a month is "1mo".

Implementing String() makes Interval satisfy fmt.Stringer, which the fmt package looks for: %v and %s on an Interval print "1h" rather than the underlying 7. Anything you would want printed nicely in a log line is worth giving a String method.

An invalid interval prints as Interval(200) rather than an empty string, so a bug shows up in the log instead of leaving a hole in it.

func (*Interval) UnmarshalText

func (i *Interval) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler, accepting either spelling.

Note the pointer receiver: *Interval, where every other method on this type takes a plain Interval. Go passes receivers by value like any other argument, so a value receiver would be handed a copy and assigning to it would change nothing observable. A method that mutates its receiver must take a pointer — and unmarshalling is mutation by definition.

type Kline

type Kline struct {
	// OpenTime is the instant the interval began, and is the candle's
	// identity: it is what deduplication, sorting and range filtering key on.
	// Always UTC.
	OpenTime time.Time `json:"open_time"`

	// CloseTime is the last instant included in the interval, as Binance
	// reports it. Note that it is inclusive and lands one millisecond (or,
	// since 2025, one microsecond) before the next candle's OpenTime, rather
	// than being equal to it.
	CloseTime time.Time `json:"close_time"`

	// Open, High, Low and Close are the first, highest, lowest and last trade
	// prices within the interval.
	Open  udecimal.Decimal `json:"open"`
	High  udecimal.Decimal `json:"high"`
	Low   udecimal.Decimal `json:"low"`
	Close udecimal.Decimal `json:"close"`

	// Volume is the quantity traded, denominated in the base asset — the BTC
	// of BTCUSDT.
	Volume udecimal.Decimal `json:"volume"`

	// QuoteVolume is the same trading denominated in the quote asset, the
	// USDT of BTCUSDT. This is the field that reaches twenty significant
	// digits and rules out float64 for the whole struct.
	QuoteVolume udecimal.Decimal `json:"quote_volume"`

	// TakerBuyBaseVolume and TakerBuyQuoteVolume are the portion of Volume and
	// QuoteVolume where the buyer was the taker — the aggressor crossing the
	// spread. The sell-side portion is the remainder, so Binance does not
	// publish it separately.
	TakerBuyBaseVolume  udecimal.Decimal `json:"taker_buy_base_volume"`
	TakerBuyQuoteVolume udecimal.Decimal `json:"taker_buy_quote_volume"`

	// Trades is the number of individual trades aggregated into this candle.
	Trades int64 `json:"trades"`
}

Kline is one candlestick: the open, high, low and close of a single interval, with the volume traded during it.

Why the numbers are not float64

Every price and volume field is a udecimal.Decimal rather than a float64, and that is the single most consequential decision in this package. Binance publishes these values as exact decimal text, and a real BTCUSDT monthly candle carries a quote volume of 118661604939.99255335 — twenty significant digits. A float64 holds about 15.95, so it cannot represent that number, and the failure is not a crash but a quiet drift in the last digits of every aggregate computed from it.

Fixed-point integers were measured and rejected too: real meme-coin volumes overflow an int64 scaled by 1e8, and Go's integer overflow is silent — the number simply comes back negative. There is no single scale that covers both BTC prices and SHIB volumes.

udecimal keeps a 128-bit coefficient inline in the struct and falls back to big.Int only past 2^128, which real Binance values never reach — so parsing the worst case measured takes 25ns and allocates nothing. The cost is that arithmetic is methods rather than operators — a.Add(b), not a + b — and that a Kline is 312 bytes rather than the 120 a float64 version measures. Convert with udecimal.Decimal.InexactFloat64 at the boundary where you genuinely need a float, which for most callers is feeding an indicator library; the column helpers below do exactly that.

docs/numbers.md has the full comparison against float64, int64 fixed point, govalues/decimal, shopspring/decimal and apd, measured over 1.75 million real archive values.

What is not in here

The Python implementation this replaces repeats the symbol and the interval on every row. They are absent here because a []Kline is already the answer to one request, for one symbol at one interval — the fields would be the same value 2.6 million times over. Go's static typing means a slice cannot quietly acquire rows from a different symbol the way a concatenated DataFrame can.

JSON field names

Every field carries a snake_case json tag, so encoding/json emits open_time rather than OpenTime. Three reasons, in order of how much they matter.

The names are the ones Binance itself uses and the ones the Python implementation this replaces wrote, so a file produced here drops into an existing pipeline without a rename step. They are also stable in a way Go field names are not: a field renamed for clarity would silently change the wire format of every consumer, and a tag makes that a deliberate act rather than a side effect. And the tags live on the type rather than on a private copy inside the CLI, because a caller marshalling a Kline in their own code should get the same document `bmd download --format json` writes.

The decimals marshal as JSON *strings*, which is udecimal's own choice and the right one: a quote volume can reach twenty significant digits, and a bare JSON number that wide loses its tail in any consumer that parses into a float64 — which is most of them, JavaScript included.

Field order

Struct fields are laid out in declaration order and padded to keep each one aligned, so declaration order is a memory-layout decision. Grouping the two 24-byte times, then the eight 32-byte decimals, then the lone int64, leaves no padding at all. Five years of one-minute candles is about 2.6 million of these, so the layout is worth one minute of thought — and is a large part of why the planned Stream API exists as an alternative to holding a whole range in memory at once.

func (Kline) Equal

func (k Kline) Equal(other Kline) bool

Equal reports whether two candles are the same in every field.

This method exists because == is a trap here, and a silent one. Go's == compares structs field by field, and it compiles for Kline — but a udecimal.Decimal holds a *big.Int pointer for values beyond 2^128, so == would compare two pointers rather than two numbers and report unequal for values that are equal. time.Time is worse: == also compares the monotonic clock reading and the *Location pointer, so the same instant read two different ways is not ==. reflect.DeepEqual has both problems too.

The correct comparisons are udecimal.Decimal.Equal and time.Time.Equal, and gathering them here means no caller has to remember that. Tests in later stages compare candles constantly; this is the function they use.

type Loader

type Loader struct {
	// contains filtered or unexported fields
}

Loader fetches candles, caching everything it downloads.

A Loader is safe for concurrent use and is meant to be long-lived: one per process, built once and shared. That is not merely convenient — the concurrency limit, the connection pool and the REST rate limiter are all per-Loader, and two Loaders each pacing themselves correctly still exceed Binance's per-IP quota together.

The zero value is not usable; build one with NewLoader.

func NewLoader

func NewLoader(opts ...Option) (*Loader, error)

NewLoader builds a Loader from the options given. See Option and the With* functions for what can be configured; the defaults are usable for everything except 1s data, where WithConcurrency wants turning down.

It performs no I/O and creates no directories. A constructor that reached for the network would make every program that builds a Loader at startup fail to start when Binance is down, and one that created the cache directory would leave a tree behind for a program that then failed validation and fetched nothing.

It does return an error, and that is the project's rule rather than this function's preference: validation that lives anywhere else is validation a caller can forget to run.

Example

ExampleNewLoader shows the options worth knowing about.

package main

import (
	"log"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	// Every option is optional and the defaults are the recommendation: the
	// cache goes in the OS cache directory, eight chunks are fetched at once,
	// and nothing is logged. Options are applied left to right, so a later one
	// overrides an earlier one.
	loader, err := binancedata.NewLoader(
		binancedata.WithCacheDir("/var/cache/backtest"),

		// Turn this *down* for 1s data. Each worker holds one decoded archive,
		// and a month of 1s candles is around 810 MB.
		binancedata.WithConcurrency(4),
	)
	if err != nil {
		// Options validate their arguments, so this reports a bad setting
		// rather than deferring it to the first fetch. The error wraps
		// ErrInvalidRequest.
		log.Fatal(err)
	}

	_ = loader
}

func (*Loader) Available

func (l *Loader) Available(ctx context.Context, q AvailabilityQuery) (Availability, error)

Available reports what Binance publishes for one symbol and interval.

It is the library half of `bmd list`, and it costs one bucket listing per granularity — two for most intervals, one for the three that have no daily archives. Nothing is downloaded and no candle is parsed.

An empty result with a nil error is the honest answer for a symbol that never traded: the bucket answered, and it said there is nothing there. That is a different outcome from an error, which means the bucket did not answer, and the two must not arrive looking the same — see the note on the bucket lister's three outcomes in docs/architecture.md.

The call takes a permit from the Loader's concurrency limit for the same reason the plan phase does: it is I/O, and a limit that only covers downloads is a limit that a hundred concurrent list calls walk straight past.

Example

ExampleLoader_Available shows asking what Binance actually publishes, rather than inferring it from a calendar.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	a, err := loader.Available(ctx, binancedata.AvailabilityQuery{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1mo,
		Market:   binancedata.MarketSpot,
		// Since bounds the answer and its cost: the bucket listing is seeked
		// with a marker built from it, so asking about 2024 onwards is one
		// round trip where asking about everything is seven for a pair that
		// has traded since 2017.
		Since: time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC),
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(len(a.Monthly), "monthly archives")
	fmt.Println(len(a.MonthlyGaps()), "holes in the middle")
	fmt.Println("REST takes over at", a.ArchivesThrough)
}

func (*Loader) CacheUsage

func (l *Loader) CacheUsage(ctx context.Context) (CacheUsage, error)

CacheUsage measures what this Loader's cache directory holds: bytes and file counts for each tier, and how much of tier 1 a prune would reclaim.

usage, err := loader.CacheUsage(ctx)
if err != nil {
    return err
}
fmt.Printf("%d bytes, %d reclaimable\n", usage.Total(), usage.Prunable)

What it costs

One pass over the cache directory, plus one open and one seek per archive to read the footer of the parquet beside it — which is what decides CacheUsage.Prunable. Nothing is hashed, nothing is decoded and nothing is downloaded, so this is orders of magnitude cheaper than Loader.VerifyCache, which reads every archive end to end.

A cache directory that does not exist yet measures zero, with no error: that is indistinguishable from a cache nothing has been written to.

func (*Loader) EvictCache

func (l *Loader) EvictCache(ctx context.Context, opts EvictOptions) iter.Seq2[EvictResult, error]

EvictCache deletes whole cache entries — archive, sidecar and parquet — selected by EvictOptions, yielding one EvictResult per entry considered:

opts := binancedata.EvictOptions{
    Symbols: []string{"BTC/USDT"},
    Before:  time.Date(2023, 1, 1, 0, 0, 0, 0, time.UTC),
}

for result, err := range loader.EvictCache(ctx, opts) {
    if err != nil {
        return err
    }
    freed += result.Size
}

Pass DryRun to reach every verdict and delete nothing.

How this differs from PruneArchives, and why they are separate

Loader.PruneArchives cannot cost you data. It removes only archives whose parquet can already serve reads on its own, so every read that succeeded before a prune succeeds after it, and the worst case is a download the day CodecVersion moves. This is the opposite operation: it removes the data itself, and every read of an evicted entry goes back to Binance.

They are two methods rather than one with more options for exactly that reason. A caller reaching for the safe one should not be one typo away from the destructive one, and a guarantee that holds for half a function's behaviour is not a guarantee anybody can rely on.

There is no automatic policy, and that is deliberate

The cache never evicts on its own — no size cap, no expiry. EvictOptions carries the measurements behind that: expiry by file age would key on when an entry was downloaded rather than when it was used, and a least-recently-used size cap needs a recency signal the filesystem does not reliably provide, which the library could only supply by writing on every cache hit. What a caller does have is knowledge of its own window — "the backtest starts at 2023 now" — and that is what this takes.

An entry is the unit, and a pruned entry is still an entry

The three files of an entry share a stem and are removed together, however many of them are on disk. This matters more than it sounds: an entry that `bmd prune` has already been over has no .zip left, and an implementation looking for archives would walk straight past exactly the entries a cache accumulates over time.

Files this library did not write are never touched, at any level — a stray file in a data directory, a directory that is not part of the layout. The name has to be one [archiveName] would have produced for the symbol, interval and granularity of the directory it sits in.

Empty directories go with the entries

A data directory emptied by an eviction is removed, and so are the parents that leaves empty, up to but not including the cache root. Nothing else collects them, and a tree of empty directories is what makes `bmd cache` report an empty cache while the layout is still visible in a file browser.

Two error channels, two meanings

The yielded error ends the iteration: the options did not validate, the cache directory could not be walked, or ctx was cancelled. A file that could not be deleted is not that and stops nothing — it arrives in EvictResult.Err with a nil error beside it, since reporting on every entry is the job.

It must not run while the same cache is being filled

The same coordination rule Loader.PruneArchives carries, and nothing enforces it here either. An entry deleted between a read establishing that its parquet is present and that read opening it turns a cache hit into a "no such file or directory" that the next call would not have. Nothing is written wrongly and nothing is lost that a download cannot replace, but it is a failure a caller did not have to have. Evict when the Loader is idle.

A cache directory that does not exist yet yields nothing and no error.

func (*Loader) Fetch

func (l *Loader) Fetch(ctx context.Context, req Request) ([]Kline, error)

Fetch returns every candle in the requested range, in ascending order of open time, with no duplicates.

The range is closed — both Start and End are included, so a candle is returned when Start <= OpenTime <= End. A full year of 2024 is Start 2024-01-01 and End 2024-12-31T23:59:59.999999999Z; see Request for why the nines are there and what writing 2025-01-01 instead would get you. A zero End means "now, as of this call".

What an error means

Nothing is returned alongside one. If any part of the range has no data in any of the three sources, the whole call fails with an error wrapping ErrNotAvailable naming the empty span, rather than returning a shorter range than was asked for. Silently returning less than requested is the failure this library is built to avoid: a backtest cannot tell the difference between "the market was quiet" and "two months are missing".

The one span exempt from that is the one that has not happened yet. A request ending now normally ends part-way through a candle that has not closed, and an unclosed candle is deliberately not returned — the candle currently forming is dropped rather than reported half-finished — so a tail with nothing settled in it is expected rather than missing.

How far that guarantee reaches

To the chunk, which is the granularity Binance publishes at. A month, day or REST range that produced nothing is an error; a chunk that produced *some* of what its span could hold is not examined further.

That line is deliberate rather than convenient. Archives are legitimately partial — SHIBUSDT's 2021-05 archive holds 22 rows for a 31-day month because the pair was listed on the 10th — so a rule that demanded a full chunk would reject real data, which is the same reason codec.go checks that every candle is inside its period but never that the period is full.

In practice the case that matters is caught anyway, because an absent period is an absent *archive*. Asking for BTCUSDT from 2015 makes every month before 2017-08 a chunk of its own with nothing in it, and that is an error. What is not caught is a pair that began trading part-way through a month whose archive does exist: the range then starts at the first real candle rather than at Start, and no error is returned.

What an error leaves behind

Possibly a warmer cache. A failing chunk cancels its siblings, but a download the cache has already started is not stopped: it finishes and populates the cache for the next run, which is the right trade for a directory that outlives the process. So this call can return an error while bytes are still being written under the cache directory, and a program that deletes that directory immediately after a failure is racing work it cannot see. Retrying, which is the ordinary response, is the case it is optimised for.

Memory

The whole range is held at once. A Kline is 312 bytes, so five years of 1m candles is roughly 820 MB. Use Loader.Stream to consume a large range without materialising it.

func (*Loader) FetchAll

func (l *Loader) FetchAll(ctx context.Context, reqs []Request) (map[Request][]Kline, error)

FetchAll runs several requests together and returns each one's candles.

The map is keyed by the request *as given*, not as resolved, so a caller looks up exactly the value they passed in. That works because Request is comparable and because its Start and End are required to be UTC — see the note on that type for why time.Time is otherwise a trap as a map key.

One budget, not one per request

Every request shares the Loader's concurrency limit, so twenty requests move as fast as the limit allows rather than twenty times faster than it. They also share the cache's deduplication: two overlapping ranges for one symbol download the months they have in common exactly once, which is what makes this a single call rather than the register-then-start API the Python implementation needed.

The first error stops everything

A failure cancels the remaining requests and FetchAll returns a nil map with that error. Returning a half-filled map alongside an error would make every caller check two things, and the useful ones would check one.

Example

ExampleLoader_FetchAll shows several requests sharing one concurrency budget.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	start := time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC)
	end := time.Date(2024, 3, 31, 23, 0, 0, 0, time.UTC)

	reqs := []binancedata.Request{
		{Symbol: "BTC/USDT", Interval: binancedata.Interval1h, Market: binancedata.MarketSpot, Start: start, End: end},
		{Symbol: "ETH/USDT", Interval: binancedata.Interval1h, Market: binancedata.MarketSpot, Start: start, End: end},
	}

	// One call rather than a goroutine per request, and the difference is not
	// only tidiness: the loader's concurrency limit spans all of them, so
	// twenty requests do not become twenty times the load. Archives two
	// requests have in common are downloaded once.
	//
	// It fails fast — the first error cancels the rest and the map comes back
	// nil, so there is no half-filled result to mistake for a whole one.
	byRequest, err := loader.FetchAll(ctx, reqs)
	if err != nil {
		log.Fatal(err)
	}

	// The map is keyed by the request itself. A Request is comparable — every
	// field is a string, a small integer or a time.Time — which is what lets
	// it be a map key at all, and it saves inventing an ID to correlate
	// results with questions.
	for _, req := range reqs {
		fmt.Println(req.Symbol, len(byRequest[req]), "candles")
	}
}

func (*Loader) PruneArchives

func (l *Loader) PruneArchives(ctx context.Context, opts PruneOptions) iter.Seq2[PruneResult, error]

PruneArchives deletes cached archives that the parquet tier no longer needs, yielding one PruneResult per archive considered:

for result, err := range loader.PruneArchives(ctx, binancedata.PruneOptions{}) {
    if err != nil {
        return err // the cache directory could not be walked
    }
    if result.Removed {
        freed += result.Size
    }
}

Pass PruneOptions with DryRun set to reach every verdict and delete nothing.

What is safe to delete, and why

Reads are served from the parquet tier: a hit reads the .CHECKSUM sidecar and the parquet's footer, and opens the archive neither to decode nor to re-hash it. An archive is therefore deletable exactly when the parquet beside it would be accepted by that read — same source hash, same CodecVersion, same schema — and this checks precisely that, with the same code the read path uses.

The guarantee that follows is worth stating plainly: **every read that succeeds before a prune succeeds after it.** Only the .zip is removed. The sidecar stays, because the hash in it is what validates the parquet, and the parquet stays because it is what answers reads.

What it costs

A download later, in the one case tier 1 is still needed: rebuilding. That happens when CodecVersion moves — the parser changed, so every cached parquet has to be built again — or when a parquet fails one of parquet's per-page checksums. Both would have been a local decode with the archive on disk and become a fetch without it.

This is why pruning is something a caller asks for and never something the cache does on its own. It is also worth knowing what there is to gain before spending that future download: tier 1 is about 40% of a cache, not most of it — measured on BTCUSDT 1m for 2024-01, 2,169,570 bytes of archive against 3,226,820 of parquet — because the tier that is read is deliberately the larger one. See CacheUsage and docs/caching.md.

Two error channels, two meanings

The yielded error ends the iteration: the cache directory could not be walked, or ctx was cancelled. An archive that was *kept* is not that and stops nothing — it arrives in PruneResult.Kept with a nil error beside it, because reporting on every archive is the whole job. A keep is not a failure either: the ordinary reason is a parquet that has not been built yet, and the archive is then the only copy of that data on the machine.

It must not run while the same cache is being filled

This is the one coordination rule on the type, and it is stated here because nothing enforces it. A prune walks and deletes with no lock: not the singleflight group Loader.Fetch collapses concurrent reads through — that group deduplicates identical work and would hand a prune somebody else's candles rather than serialise against them — and not a lock file, so a second process running `bmd prune` is outside any guard this one could take.

What that costs is a race window rather than corruption. The read path establishes that tier 1 is present and then opens it; an archive deleted between those two steps turns a rebuild that would have decoded from disk into "cache: opening BTCUSDT-1h-2024-01.zip: no such file or directory", where a tier 1 that had been absent all along would simply have been downloaded. Nothing is lost and nothing is written wrongly — the call fails and the next one succeeds — but it is a failure a caller did not have to have.

So: prune when the Loader is idle, and do not run `bmd prune` against a cache directory a download is writing into.

A cache directory that does not exist yet yields nothing and no error.

func (*Loader) Stream

func (l *Loader) Stream(ctx context.Context, req Request) iter.Seq2[Kline, error]

Stream yields the requested candles one at a time, in ascending order of open time, without holding the whole range in memory.

Ranging over it is the intended use, and `break` is honoured — the pipeline behind it is cancelled and every worker stops:

for k, err := range loader.Stream(ctx, req) {
    if err != nil {
        return err
    }
    ...
}

What it costs and what it saves

Chunks are fetched concurrently and yielded in order, and a worker holds its permit until its candles have been consumed. So the memory in flight is bounded by the concurrency limit rather than by the length of the range: five years of 1m candles streams in about 110 MB instead of 820 MB.

The floor is one chunk, and a chunk is a whole archive. A month of 1s candles decodes to roughly 810 MB whatever this function does, because the cache's unit is a file. Streaming bounds how many of those exist at once; it does not make one of them smaller.

Errors

An error is yielded once, with the zero Kline, and the iteration then ends. Unlike Loader.Fetch it may arrive *after* some candles have already been yielded — a stream cannot un-yield what the caller has already seen — so a consumer that needs all-or-nothing should use Fetch.

Example

ExampleLoader_Stream shows how to consume a range too large to hold in memory.

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"github.com/quagmt/udecimal"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	req := binancedata.Request{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1m,
		Market:   binancedata.MarketSpot,
		Start:    time.Date(2020, 1, 1, 0, 0, 0, 0, time.UTC),
		// A zero End means "now, resolved when this call runs". Prefer it to
		// writing time.Now(): a stored end date is a snapshot that ages, and
		// having nothing stored is the whole point of the zero value.
	}

	// Stream returns an iter.Seq2, which `range` consumes directly — this is
	// Go's range-over-function, and the pair yielded is (value, error) rather
	// than (index, value). Five years of 1m candles is about 820 MB as a
	// slice; streamed, the memory in flight is bounded by the concurrency
	// limit instead, at roughly 110 MB.
	var high udecimal.Decimal

	for k, err := range loader.Stream(ctx, req) {
		if err != nil {
			// The error is yielded, not returned, so it has to be checked
			// inside the loop. Returning or breaking here cancels the
			// pipeline behind the iterator and stops every worker.
			log.Fatal(err)
		}

		if k.High.GreaterThan(high) {
			high = k.High
		}
	}

	fmt.Println("highest price:", high)
}

func (*Loader) VerifyCache

func (l *Loader) VerifyCache(ctx context.Context) iter.Seq2[CacheEntry, error]

VerifyCache re-hashes every archive in this Loader's cache against the .CHECKSUM sidecar Binance published with it, yielding one CacheEntry per archive:

for entry, err := range loader.VerifyCache(ctx) {
    if err != nil {
        return err // the cache directory could not be walked
    }
    if entry.Err != nil {
        fmt.Println(entry.Path, entry.Err)
    }
}

It is the on-demand half of the library's integrity guarantee. Archives are verified once, when they are downloaded, and never again: re-hashing a 93 MB file on every read would cost more than the CSV parse the second cache tier exists to avoid. That leaves one gap — a file that was correct when written and was damaged afterwards — and this is how it is closed, whenever a caller decides to spend the I/O.

Two error channels, two meanings

The yielded error ends the iteration: the cache directory could not be walked, or ctx was cancelled. A bad *archive* is not that and does not stop anything, because reporting every bad file is the whole job — it arrives in CacheEntry.Err with a nil error beside it. This is the same split Loader.Stream uses, and the loop above is the shape both want.

Nothing is deleted, downloaded or repaired. What to do about a mismatch is the caller's decision, and the cache heals itself either way: an archive that is removed is downloaded again on the next request for it.

A cache directory that does not exist yet yields nothing and no error, since that is indistinguishable from a cache with no files in it.

Example

ExampleLoader_VerifyCache shows re-hashing every cached archive against the checksum Binance published with it.

package main

import (
	"context"
	"fmt"
	"log"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	loader, err := binancedata.NewLoader()
	if err != nil {
		log.Fatal(err)
	}

	// An iterator rather than a slice, because a large cache takes a while to
	// hash and a caller wants to report each file as it lands. The outer error
	// is for a failure to walk the cache at all; a single bad archive arrives
	// as CacheEntry.Err with a nil outer error, so one corrupt file does not
	// abandon the scan.
	bad := 0

	for entry, err := range loader.VerifyCache(ctx) {
		if err != nil {
			log.Fatal(err)
		}

		if entry.Err != nil {
			bad++

			fmt.Printf("%s: %v\n", entry.Path, entry.Err)
		}
	}

	fmt.Println(bad, "archives failed verification")
}

type Market

type Market uint8

Market selects which Binance market a request refers to.

Only MarketSpot is implemented. The type exists anyway, ahead of any second value, because it is one of three deliberate extension points that make adding futures an increment rather than a rewrite: every URL this library builds passes through a switch on a Market, so a new market is a new case in a handful of switches the compiler will point at. Threading a bare "spot" string through instead would leave nothing to point at.

The `exhaustive` linter (see .golangci.yml) enforces that: adding a constant to this type turns every switch that has not been updated into a lint failure, which is as close as Go gets to a compile-time reminder.

const (
	MarketSpot Market = iota + 1 // spot market — the only implemented value
)

The supported markets.

As with Interval, the `+ 1` leaves 0 unassigned so that the zero value — what an unset struct field holds — is an invalid market rather than a plausible one. A caller must name the market they want.

Making spot the zero value would have been friendlier today and wrong tomorrow. Spot is the only market now, so an unset field could only ever have meant spot; but the moment futures exists, every request written before it existed silently keeps meaning spot, and the one place the compiler could have asked "which market?" has been given away permanently. A default is a decision you can only make once, and it is much easier to add later than to take back.

The rule underneath: Go gives every variable its zero value and no constructor can intercept that, so deciding what your zero value means is not optional. You either choose it or inherit it by accident. This codebase chooses the same thing every time — the zero value is invalid, and being explicit is the price of entry.

func ParseMarket

func ParseMarket(s string) (Market, error)

ParseMarket converts a market name into a Market. It accepts exactly the names Market.String produces, so the CLI's --market flag and the log lines describing what it did use one vocabulary.

The returned error wraps ErrInvalidRequest.

func (Market) IsValid

func (m Market) IsValid() bool

IsValid reports whether m is a market this library supports.

func (Market) MarshalText

func (m Market) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler. See Interval.MarshalText.

func (Market) String

func (m Market) String() string

String returns the market's name as it appears in flags and log lines. It also happens to be the path segment data.binance.vision uses for spot, though that correspondence will not survive futures — /data/futures/um/ is two segments — so URL building gets its own mapping when it arrives in Stage 4.

This implements fmt.Stringer; see Interval.String for what that buys.

func (*Market) UnmarshalText

func (m *Market) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler. The pointer receiver is required because the method assigns to its receiver; see Interval.UnmarshalText.

type Option

type Option interface {
	// contains filtered or unexported methods
}

Option configures a Loader. See NewLoader and the With* functions below.

An Option is something you receive from a With* function and hand to NewLoader. There is nothing else to know about it, and nothing outside this package can build one — which is the whole reason it is an interface.

Why an interface, when an option is plainly a function

The obvious spelling is a named function type, and that is what this was until Stage 9:

type Option func(*loaderConfig) error

It compiles to the same thing and callers write the same code. What it also does is publish a signature naming loaderConfig — a type no other package can see, let alone write down — so the generated documentation renders the declaration as an instruction the reader is unable to follow. The private configuration struct leaks out through the one place it was supposed to stay behind.

An interface whose only method is unexported renders instead as a named type with its contents filtered out, which is honest rather than teasing. It also makes the type closed by construction: apply is unexported, so no other package can satisfy Option even by accident, and this package stays free to add a method to it later without breaking a single caller.

This is the same reasoning that kept the domain types out of internal/ in Stage 2. An identifier the documentation names but the reader cannot reach is worse than one it never mentions.

func WithCacheDir

func WithCacheDir(dir string) Option

WithCacheDir sets the directory holding cached archives and their derived Parquet files.

The default is a "bmd" directory inside the operating system's own cache location — ~/Library/Caches on macOS, $XDG_CACHE_HOME or ~/.cache on Linux, %LocalAppData% on Windows. That is the right default because everything in it is re-downloadable by definition and none of it should ever be backed up.

The directory is created on first write, not here and not by NewLoader: a loader that only ever gets cache hits, or whose requests all fail validation, leaves nothing behind.

An empty dir is rejected rather than treated as "use the default". The two spellings would be indistinguishable at the call site, and a configuration file with a missing key would silently write to somewhere its author did not choose.

No environment variable is consulted here, and that is deliberate rather than an omission. This package is imported by other programs, and one that read the environment on its own would let a variable exported for some unrelated reason redirect where its caller writes files. The bmd command does read one — BMD_CACHE_DIR, which its -cache-dir flag overrides — because a tool a person runs is the layer where that is the expected behaviour rather than a surprise.

func WithConcurrency

func WithConcurrency(n int) Option

WithConcurrency sets how many chunks are fetched at once. The default is 8.

One number governs the whole loader, not one per call: Loader.FetchAll running twenty requests uses the same budget as a single Loader.Fetch, so the setting means what it says regardless of how the work arrives.

Turn it *down* for 1s data. Each worker holds one decoded archive, and a month of 1s candles is around 810 MB, so the default of 8 is several gigabytes at that interval. The default is chosen for 1m and coarser, where it costs at most about 110 MB.

func WithHTTPClient

func WithHTTPClient(client *http.Client) Option

WithHTTPClient supplies the http.Client used for every request.

The default is a package-wide client built in internal/vision, and it is worth knowing what you are replacing before you replace it. It clones http.DefaultTransport — keeping proxy support and HTTP/2 negotiation, which a hand-built &http.Transport{} silently drops — and raises MaxIdleConnsPerHost from its default of 2 to 64, because every request this library makes goes to one of three hosts. Passing http.DefaultClient here means two idle connections per host, and a worker pool that re-handshakes TCP and TLS for most of its downloads.

It also deliberately sets no Client.Timeout. A timeout bounds the entire exchange including the body read, so any value large enough for a 93 MB archive on a slow link is far too large to catch a hung connection. Use the context instead: it is per call and the caller owns it.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the structured logger. The default discards everything.

The loader logs at two levels and nothing else writes: Debug for each decision that is invisible from the outside — a chunk the bucket listing said was missing, an archive that 404'd anyway, a fallback down the ladder — and Warn for the pipeline pausing on a rate limit. Errors are returned rather than logged, since a library that logs an error and returns it has reported it twice.

func WithProgress

func WithProgress(fn func(Progress)) Option

WithProgress registers a function called once for each chunk of work that finishes. See Progress for what it is told and what it is not.

Calls are serialised: the loader holds a mutex across the call, so fn is never entered by two goroutines at once and does not need to be safe for concurrent use. That is a promise worth making explicitly, because the obvious implementation — call it from each worker — would quietly require every caller to write a mutex of their own, and most would not.

The cost of that promise is that fn is on the critical path: a slow callback stalls the pool. Keep it to a counter, a progress bar or a channel send.

Example

ExampleWithProgress shows reporting progress while a long range downloads.

package main

import (
	"context"
	"log"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	ctx := context.Background()

	// The callback is serialised — the loader holds a mutex across it — so it
	// does not need to be safe for concurrent use. The cost of that promise is
	// that it sits on the critical path, so keep it to a counter, a progress
	// bar or a channel send.
	onProgress := func(p binancedata.Progress) {
		if p.Err != nil {
			// Reported here *and* returned from the call. This says which unit
			// of work failed while the run was still going; the returned error
			// is what to act on.
			log.Printf("chunk failed: %v", p.Err)

			return
		}

		log.Printf("%d/%d  %s  %s  %d candles",
			p.Done, p.Total, p.Source, p.Start.Format(time.DateOnly), p.Klines)
	}

	loader, err := binancedata.NewLoader(binancedata.WithProgress(onProgress))
	if err != nil {
		log.Fatal(err)
	}

	_, err = loader.Fetch(ctx, binancedata.Request{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1m,
		Market:   binancedata.MarketSpot,
		Start:    time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC),
		End:      time.Date(2024, 12, 31, 23, 59, 0, 0, time.UTC),
	})
	if err != nil {
		log.Fatal(err)
	}
}

func WithRateLimit

func WithRateLimit(weightPerSecond float64) Option

WithRateLimit lowers the sustained rate this loader is allowed to spend against the REST endpoint's quota, in weight units per second. The default is vision.DefaultWeightPerSecond.

What the unit is

Binance meters the REST mirror as REQUEST_WEIGHT, not as requests: 6000 per minute per IP address, measured on 2026-08-20, and one klines call costs 2 of them. So the quota is 100 weight per second, the default takes 40, and the argument here is in the same unit Binance publishes rather than a translation of it that would need re-deriving every time the cost of a call changes.

Only the REST tail is paced. data.binance.vision is a static file server with no quota, so listing and archive downloads are unaffected — WithConcurrency is the knob for those.

Why you would turn it down

The quota is per IP, not per API key or per process, so everything on the machine draws from the same 6000: a live trading bot, a second backtest, another copy of this library. The default already leaves most of the budget unspent for exactly that reason, and it cannot know how much company it has. If a history download shares an address with something latency-sensitive, spending less than 40 is the way to say so.

Exceeding the quota is worse than being slow. Binance escalates a 429 a client keeps ignoring into an HTTP 418, an IP ban running from two minutes to three days and lengthening with repeat offences — which punishes the address rather than the process, so the ban shows up in the trading bot's logs.

Why it will not let you go up

Values above the quota's own 100 per second are rejected rather than clamped. There is no rate above it that Binance permits, so accepting one would be accepting a setting that cannot be honoured, and clamping silently would have this function report success for a policy it did not apply. The ceiling is the quota itself rather than the default, so raising the rate towards it stays possible for a caller who knows they have the address to themselves.

The one thing to know before using it

A loader built without this option shares one process-wide limiter with every other such loader, because two buckets each honouring the documented rate permit twice it — correct alone and wrong in aggregate. This option opts out of that sharing: the loader gets a bucket of its own. With a single loader in the process, which is the normal case, that is exactly what it looks like. With several, set it on all of them, or the ones left on the default are spending from a second bucket that knows nothing about this one.

type Progress

type Progress struct {
	// Request is the request this work belongs to, in resolved form: the
	// symbol normalised and End filled in if it was left zero. In FetchAll
	// this is what identifies which of the requests a callback is about.
	Request Request

	// Source is where the candles came from.
	//
	// It names what the *plan* asked for. If the archive turned out to be
	// missing and the candles were recovered from further down the ladder,
	// this still says what was planned, and the substitution is reported to
	// the logger instead. One event per planned unit of work is what keeps
	// Done and Total meaningful.
	Source Source

	// Start and End are the half-open range the chunk covered — Start
	// included, End excluded. For an archive this is the archive's own
	// extent, which is routinely wider than the part of it the request wanted.
	//
	// Note the mismatch with the Request above, and that it is deliberate.
	// A [Request] is closed: the caller's End is included. A chunk is
	// half-open, because chunks are the pieces a range is cut into and
	// half-open pieces join without arithmetic — see [Request] on where each
	// convention lives. These fields describe the pieces, so they use the
	// pieces' convention, and a display that prints them alongside the
	// request's own range should say so rather than let a reader assume the
	// last candle of the chunk opened at End.
	Start, End time.Time

	// Klines is how many candles the chunk produced, before merging and
	// trimming. Zero is normal at the leading edge of a symbol's history and
	// for a range that is still forming; it is not normal in the middle of a
	// published month, and the loader turns that case into an error rather
	// than a quiet gap.
	Klines int

	// Done and Total count chunks, not bytes: Done is how many have finished
	// including this one, Total how many the plan holds. Total is fixed before
	// any work starts, so Done/Total is a genuine fraction rather than an
	// estimate that moves.
	Total int
	Done  int

	// Err is the error the chunk failed with, or nil.
	//
	// A failure is reported here *and* returned from the call, because the two
	// answer different questions: this one says which unit of work failed
	// while the run was still going, and the returned error is what the caller
	// acts on. A progress callback is not a substitute for checking the error.
	Err error
}

Progress describes one finished chunk of work, and is what WithProgress receives.

What it does not say

There is no "cache hit" field, and its absence is a design decision rather than an oversight. The cache's entire surface within this package is one method that takes an archive and returns its candles; which tier answered, whether anything was downloaded and whether the Parquet file had to be rebuilt are deliberately invisible above that line, and adding them to this struct would mean widening that surface for the sake of a progress bar. See docs/caching.md.

What is here is the shape of the work — how many units, how far through, and how much each one yielded — which is what a progress display and a diagnostic log actually need.

type PruneOptions

type PruneOptions struct {
	// DryRun decides everything and deletes nothing. Every [PruneResult] comes
	// back with the verdict it would have acted on, and Removed false
	// throughout.
	DryRun bool
}

PruneOptions controls one prune.

It is a struct rather than a bare parameter for the reason AvailabilityQuery is: a call reading PruneOptions{DryRun: true} says at the call site what prune(ctx, true) would leave the reader to look up — and this is a call whose second argument decides whether files are deleted.

type PruneResult

type PruneResult struct {
	// Path is the archive's absolute path.
	Path string

	// Size is its size in bytes — what pruning it reclaims, and what it still
	// occupies if it was kept.
	Size int64

	// Kept says why this archive was not pruned, and is nil when it was
	// prunable.
	//
	// A non-nil value is not a failure. The ordinary cause is an archive whose
	// parquet has not been built yet, or was built by an older [CodecVersion]:
	// in both cases tier 1 is the only copy of that data there is, and deleting
	// it would turn a rebuild into a download.
	Kept error

	// Removed reports whether this call deleted the file. It is false whenever
	// Kept is non-nil, and false in a dry run even for an archive that was
	// prunable — which is why a caller totalling "what would this free" must
	// count Kept == nil rather than Removed.
	Removed bool

	// Err is set when the archive was prunable and deleting it failed. The
	// archive is still there; nothing else about the cache changed.
	Err error
}

PruneResult is one archive a prune considered, and what it decided.

The three outcomes are distinguishable without comparing anything to a sentinel: Kept non-nil means the archive is still needed, Removed true means it is gone, and Err non-nil means it should have gone and would not.

type Request

type Request struct {
	// Symbol is the trading pair, in any of the spellings [NormalizeSymbol]
	// accepts: "BTC/USDT", "BTC-USDT" or "BTCUSDT".
	Symbol string

	// Interval is the candle aggregation period. Required; the zero value is
	// invalid.
	Interval Interval

	// Market selects the Binance market. Required; the zero value is invalid.
	// [MarketSpot] is the only implemented value.
	Market Market

	// Start is the first instant included in the range. Required, and must be
	// UTC.
	Start time.Time

	// End is the last instant *included* in the range, and must be UTC. A
	// candle is returned when its open time is at or before it.
	//
	// The zero value means "now, as of the moment the request is executed".
	// Prefer leaving it zero over writing time.Now(): a stored End is a
	// snapshot that ages, and this field exists to not have one.
	End time.Time
}

Request describes one range of candles to fetch: which symbol, at which interval, in which market, over which span of time.

Closed ranges

Both ends are included — the range mathematicians write [Start, End]. A candle is in the range when

Start <= OpenTime <= End

so End is most usefully read as *the open time of the last candle you want*. For daily candles, End 2024-03-31 returns the candle for the 31st. For hourly candles, End 2024-03-31 returns the one candle that opened at 00:00 on the 31st — because that is the last candle whose open time is at or before the instant named. Ask for the whole day and you are asking for the whole day's last instant: 2024-03-31T23:59:59.999999999Z. The bmd CLI does that expansion for you when --end is written as a bare date, which is the right place for a human-facing convenience.

Where the half-open ranges went

They are still here; they are just no longer the caller's problem. Inside the pipeline every boundary is half-open, because ranges are split into months, months into days, days into API pages, and the pieces must join back together with no arithmetic at all:

[Jan 1, Feb 1) + [Feb 1, Mar 1)  =  [Jan 1, Mar 1)

The end of one piece *is* the start of the next, so a seam is written once and there is nothing to add or subtract. Inclusive seams would need a "+1 of something" at every join, and the something changes with the interval — one millisecond before 2025, one microsecond after. Every one of those is a chance to drop a candle or emit it twice, silently.

So the conversion from what a caller wrote to what the pipeline uses happens exactly once, in a single unexported method, and the something it adds is one *nanosecond*. That is a unit no Binance timestamp has ever used: the archives publish milliseconds, and microseconds since 2025. Nothing can fall between End and End+1ns, so the conversion cannot gain or lose a candle, and it is the same single line whichever side of the 2025 switch the data sits on.

What this costs, stated plainly

A whole year of 2024 is now

Start: 2024-01-01T00:00:00Z, End: 2024-12-31T23:59:59.999999999Z

Writing End 2025-01-01 instead is not an error and will not be reported as one. It asks for the candle that opens exactly at midnight on New Year's Day, which is a real candle living in a different month — so the planner fetches January's archive to get it, and one extra candle arrives at the end of your slice. That is the tax inclusive ends charge, and it is charged to whoever writes the boundary. It is here because the alternative was worse: a CLI whose --end meant something different from the library's End is the kind of difference nobody notices until a backtest is a day short.

Zero values mean something here

Go gives every struct field a zero value and provides no constructors, so a caller can always write Request{} and there is nothing this package can do to stop them. The defence is to make the zero value of each field either meaningful or detectably invalid:

• Interval and Market number their constants from 1, so a zero field is an invalid value rather than a plausible one that silently defaults.
• Start must be set; a zero Start is rejected.
• End is the exception: a zero End means "up to now", resolved at the moment of the call.

That last one is deliberate, and it is a bug fixed rather than ported. The Python implementation defaulted its end date with datetime.now(UTC) evaluated as a *default argument*, which Python binds once when the module is imported. A process that runs for a week therefore keeps asking for data up to the day it started, and quietly returns less and less of what was requested. Storing the zero value and resolving it per call cannot drift, because there is nothing stored to go stale.

On using Request as a map key

FetchAll returns map[Request][]Kline, which requires Request to be comparable — it is, since every field is. But two of those fields are time.Time, and time.Time equality under == is not what you want: it compares the wall clock, the monotonic reading and the *time.Location pointer. Two times naming the same instant in different locations are not == to each other, and a time from time.Now() carries a monotonic reading that one read back from a database does not.

Requiring Start and End to be UTC is what makes the map key safe. UTC is a single shared Location value, and the .UTC() conversion that produces it also strips the monotonic reading. So the rule below is not pedantry about time zones; it is what stops FetchAll from returning two entries for one request.

func (Request) Validate

func (r Request) Validate() error

Validate reports whether the request is well-formed, without consulting a clock or the network. It returns an error wrapping ErrInvalidRequest, or nil.

Every check here is one that can be made before a single byte is sent, which is the point: a request that cannot possibly succeed should fail immediately and cheaply, naming the field at fault, rather than 404-ing several layers away where the cause is no longer obvious.

A request with a zero End passes validation — that spelling is legal and means "up to now". What Validate cannot check is whether Start precedes an End that does not exist yet; that comparison happens when the range is resolved against a clock.

Calling this is optional. Loader.Fetch runs the same checks itself, because validation a caller has to remember to invoke is validation that eventually does not run. It is exported so that a program which builds requests from user input — a config file, CLI flags, a web form — can reject bad ones at the edge, where it still has the context to say which line was wrong.

Example

ExampleRequest_Validate shows a request being rejected before any I/O.

package main

import (
	"errors"
	"fmt"
	"time"

	binancedata "github.com/algo-one/binance-data-downloader"
)

func main() {
	req := binancedata.Request{
		Symbol:   "BTC/USDT",
		Interval: binancedata.Interval1h,
		Market:   binancedata.MarketSpot,
		Start:    time.Date(2024, 3, 1, 0, 0, 0, 0, time.UTC),
		// Local time, not UTC. Every instant in this library is UTC, and a
		// zone that happens to be UTC+0 today is not the same promise.
		End: time.Date(2024, 3, 2, 0, 0, 0, 0, time.Local),
	}

	err := req.Validate()
	fmt.Println(errors.Is(err, binancedata.ErrInvalidRequest))

	// Fetch calls Validate itself, so this method is for a caller who wants to
	// check a request early — while a form is being filled in, or before
	// queueing a batch — rather than at the point of use.
	req.End = time.Date(2024, 3, 2, 0, 0, 0, 0, time.UTC)
	fmt.Println(req.Validate())

	// A single instant is a legal request for exactly one candle: the range is
	// closed, so Start and End are both included.
	req.End = req.Start
	fmt.Println(req.Validate())

}

Output:
true
<nil>
<nil>

type Source

type Source uint8

Source says where a chunk of candles came from.

It mirrors the internal planner's own enumeration rather than exposing it. internal/plan cannot appear in this package's public API — consumers cannot import an internal package, so a public function returning plan.Kind would name a type they are unable to write down — and the indirection is cheap: one type and one switch, in exchange for the planner staying free to change.

const (
	SourceMonthlyArchive Source = iota + 1 // one ZIP covering a calendar month
	SourceDailyArchive                     // one ZIP covering a single day
	SourceRESTAPI                          // paginated calls to the REST API
)

The three places a candle can come from. Numbered from 1 so that the zero value is invalid rather than a plausible default, as with Interval and Market.

func (Source) String

func (s Source) String() string

String implements fmt.Stringer.