Documentation
¶
Overview ¶
Package plan turns a time range into the list of downloads that covers it.
Why this is its own package ¶
Every calendar rule in this library lives here, and nothing else does. The package imports "errors", "fmt" and "time" — that is the complete list, and it is enforced by the compiler rather than by discipline. A package that does not import net/http cannot make a network request by accident, however carelessly someone edits it later.
That guarantee is worth a small amount of awkwardness. The domain types (binancedata.Interval and friends) live in the root package, and the root package imports this one, so this package cannot import the root back — Go forbids import cycles outright. The workaround is Spec, which carries the two or three facts about an interval this package actually needs as plain booleans. It is a little clumsy to fill in, and in exchange the calendar logic is testable with no clock, no network, and no fixtures.
What a plan is ¶
Binance publishes historical klines in three places, and a range of any length usually needs all three:
monthly archive one ZIP per calendar month the bulk of any range daily archive one ZIP per day whole months not yet published REST range paginated API calls the last day or two
Expand does the decomposition assuming every archive it names exists. Whether they actually exist is a question for the network, so it is asked later; Substitute is the pure rule for what to do with the answer.
The invariant that matters ¶
The chunks Expand returns are sorted, contiguous, and cover the whole requested range. Contiguous means each chunk begins exactly where the previous one ended — not a millisecond later — so every instant in the range belongs to exactly one chunk. Chunks may extend *past* the requested range at either end, because archives are whole days and whole months and cannot be cut in half; the reduce step trims the result.
Expand checks this itself before returning, on every call. The check is a single pass over a handful of chunks, so it costs nothing measurable, and it converts the one failure mode nobody would notice — a missing day in the middle of a range, returned with no error — into a loud one. The implementation this library replaces had two such gaps.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrCoverageGap = errors.New("plan does not cover the requested range")
ErrCoverageGap reports that Expand produced chunks which do not cover the requested range without gaps or overlaps.
This is an assertion failure, not a condition any input should produce. It exists because the alternative to checking is not "no bug", it is "a silent bug": a range missing its last two days looks exactly like a range that had no data for its last two days. If this error ever escapes, the arithmetic in this file is wrong and the stack trace points at it.
var ErrInvalidSpec = errors.New("invalid plan spec")
ErrInvalidSpec reports that a Spec describes something impossible: a range that ends before it starts, times that are not UTC, bounds that are not aligned to midnight.
A caller of this package cannot cause this — the root package validates the user's request long before building a Spec — so in practice it means the wiring between the two is wrong. It is a real error rather than a panic because a library should not take a process down over its own bug.
Functions ¶
This section is empty.
Types ¶
type Chunk ¶
Chunk is one unit of work: a source and the half-open range it covers.
For an archive chunk the range is the archive's own extent — a whole calendar month or a whole day — regardless of how much of it the request wanted, so a chunk routinely covers more than was asked for and the reduce step trims the result.
How much more ¶
A month that is wanted in full, or wanted at an interval with no daily archives, is always one monthly chunk. The interesting case is a month that is published in full and wanted in part, and there the answer is a trade rather than a principle:
requests bytes 31 daily archives 62 (a zip and a sidecar each) only the days wanted 1 monthly archive 2 the whole month
Neither end of that is right for both shapes of request. Twenty-five days of January as twenty-five daily downloads is fifty requests to avoid fetching six days nobody asked for, and it leaves the cache holding twenty-five files that the next request for January cannot use as a month. One day of January as a monthly download is 93 MB of 1s candles to serve 86,400 of them.
So the rule is a threshold: take the month once at least half of it is wanted, and take days otherwise. Over-fetching really is cheap in a cache-backed library — it is the next eleven requests already answered — but "cheap" is a claim about the ratio between what was fetched and what was wanted, and that ratio is what the threshold bounds.
The rule lives in Consolidate rather than in Expand, because it is only sound when the monthly archive actually exists, and only the bucket listing knows that.
func Consolidate ¶
Consolidate replaces runs of daily chunks with the monthly archive covering them, wherever that month exists and enough of it is wanted.
monthExists is asked about the first instant of a month and must answer from the bucket listing. It is a parameter rather than a lookup because this package has no network and is not allowed one — see the package comment — so the caller supplies the one fact the trade-off turns on.
Why this is not part of Expand ¶
The threshold trades requests against bytes: one monthly download instead of up to sixty-two daily ones, at the cost of fetching days nobody asked for. That trade is only worth making if the monthly archive is there. If it is not, the chunk 404s and Substitute fans it back out into *every* day of the month rather than the days that were wanted — strictly worse than never having consolidated, and worse than the plan before the threshold existed.
This used to be decided inside Expand, against Spec.ArchivesThrough, which cannot answer the question: it is the later of the monthly and daily frontiers, so for most of every month it says "published" about a month whose archive Binance has not written yet. Daily archives lag real time by about a day and monthly ones by up to a month plus that day, so the window in which the two disagree is not an edge case — it is most of the time.
What it will not do ¶
Cross a month boundary, reorder anything, or touch a chunk that is not a daily archive. Runs are maximal within one calendar month, and the monthly chunk that replaces one covers at least the run's own span, so coverage is preserved — wider, never narrower, which is what Chunk documents as normal.
func Expand ¶
Expand decomposes a range into the chunks that cover it, assuming every archive it names exists.
The returned chunks are sorted, contiguous, and cover [Start, End) — see the package comment for what that guarantees and why it is checked rather than asserted in prose. Chunks may begin before Start or end after End, because archives are indivisible.
Availability is not consulted here, because consulting it needs the network and this package does not have one. The caller probes the chunks it gets back and hands any that turn out to be missing to Substitute.
Example ¶
ExampleExpand doubles as documentation: `go test` runs it and compares stdout against the Output comment, so the example on pkg.go.dev cannot drift out of date the way a hand-written snippet in a doc comment can.
chunks, err := Expand(Spec{
Start: time.Date(2026, 7, 28, 0, 0, 0, 0, time.UTC),
End: time.Date(2026, 8, 18, 0, 0, 0, 0, time.UTC),
ArchivesThrough: time.Date(2026, 8, 17, 0, 0, 0, 0, time.UTC),
HasDaily: true,
HasMonthly: true,
})
if err != nil {
panic(err)
}
fmt.Println(len(chunks), "chunks")
fmt.Println("first:", chunks[0])
fmt.Println("last: ", chunks[len(chunks)-1])
Output: 21 chunks first: daily[2026-07-28T00:00:00Z,2026-07-29T00:00:00Z) last: rest[2026-08-17T00:00:00Z,2026-08-18T00:00:00Z)
func Substitute ¶
Substitute returns the chunks to try in place of one that turned out not to exist, or an error if there is nothing left to try.
Absence is normal. Binance has published archives with holes in them — the 1mo archive for BTCUSDT is missing March 2024 while February and April are both present, verified against the live bucket on 2026-08-18 — and no amount of date arithmetic predicts that. The fallback ladder is:
monthly archive ──► daily archives ──► REST range
(skipped when the interval has none)
The returned chunks are contiguous and cover *at least* the range passed in, so a caller can splice them in where the chunk used to be. At least, rather than exactly, for the reason Chunk gives: a daily archive is a whole day and cannot be cut, so replacing a chunk that begins at 07:00 means beginning at the midnight before it. Substituting is therefore the one operation that can make a plan overlap itself, and the reduce step — which deduplicates on open time — is what absorbs that.
type Kind ¶
type Kind uint8
Kind is which of the three sources a chunk will be fetched from.
Like the enumerations in the root package it counts from 1, so that the zero value of a Chunk is detectably unset rather than silently meaning "monthly".
type Spec ¶
type Spec struct {
// Start and End are the half-open range being requested, both UTC. They
// come from a resolved binancedata.Request, so End has already had "now"
// substituted for the caller's zero value — this package never asks what
// time it is.
Start time.Time
End time.Time
// ArchivesThrough is the first instant no bulk archive covers: everything
// at or after it must come from the REST API. It must be midnight UTC,
// because archives are whole days and whole months.
//
// This is a fact about what Binance has published, discovered by listing
// the bucket, and it is a parameter rather than a calculation on purpose.
// The obvious shortcut is to assume archives lag real time by a day and
// subtract; that is a guess, and a guess that is wrong for one day drops
// that day without saying so. Asking the bucket what exists costs one HTTP
// request and removes the question.
//
// The zero value means no archives exist at all, and the whole range
// resolves to REST. That is the right answer for a symbol listed this
// week, and it falls out of the arithmetic rather than needing a case.
ArchivesThrough time.Time
// HasDaily and HasMonthly are binancedata.Interval.HasDailyArchives and
// HasMonthlyArchives for the interval being requested. Passing the two
// answers rather than the Interval is what keeps this package free of the
// import cycle described in the package comment.
//
// Only three intervals differ: 3d, 1w and 1mo have monthly archives but no
// daily ones, their candles being longer than a day.
HasDaily bool
HasMonthly bool
}
Spec is everything Expand needs, in types this package can hold without importing the root package. See the package comment for why that constraint exists.
Source Files