ursus

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 29 Imported by: 0

README

preamble from human author

this project is not production ready, and currently it is noticeably slower than other similar tools, like duckdb go or polars py.

ursus

A Polars-class dataframe library for Go 1.27 — lazy execution with a query optimizer, Arrow memory layout, SIMD kernels, and streaming execution that spills to disk rather than falling over.

df, err := ursus.ScanParquet("events.parquet").
    Filter(ursus.Col("price").Gt(5)).
    GroupBy(ursus.Col("region")).
    Agg(
        ursus.Col("price").Sum().Alias("revenue"),
        ursus.Col("qty").Mean().Alias("avg_qty"),
    ).
    Sort(ursus.Desc(ursus.Col("revenue"))).
    Collect(ctx)

The projection reaches the Parquet reader, the filter becomes a row-group predicate, and the group-by runs on every core. None of that is visible in the query.

Install

go get github.com/advenn/ursus

Go 1.27 or newer is required. Two of its changes are load-bearing: generic methods are why Series[T].Map[U] and df.Column[T](name) exist at all, and because a generic method still cannot satisfy an interface, every public type is a concrete struct with polymorphism kept in unexported interfaces.

GOEXPERIMENT=simd is optional. It switches on the SIMD kernels:

GOEXPERIMENT=simd go build ./...

Without it every kernel falls back to its scalar twin, behind //go:build !(goexperiment.simd && amd64). That is not a claim — CI runs the entire suite with the experiment off on every push, and make test-all includes an experiment-off leg locally. The flag buys speed, not correctness.

Status

v0.2 is complete. What works today:
Sources Parquet and CSV (read and write), in-memory frames
Types Bool, Int8–64, Uint8–64, Float32/64, String, Binary, Date, Time, Datetime (unit + zone), Duration, Decimal (128-bit), Enum
Expressions arithmetic, comparison, Kleene three-valued logic, conditionals, casts, null repair, .str and .dt namespaces, 19 aggregates, window functions
Frame ops filter, select, with-columns, sort, top-k, distinct, concat/vstack/hstack, slice/tail/reverse/row-index, drop/rename/drop-nulls
Joins all seven equi-join kinds with Validate, as-of join with tolerance and by keys, merge-sorted
Grouping group-by, GroupByDynamic, Rolling, calendar-aware intervals
Optimizer predicate pushdown (including through joins), projection pushdown, limit/top-k pushdown, constant folding and expression simplification
Execution order-preserving pipeline parallelism, parallel hash aggregation, and spilling for sort, hash aggregation and hash join

Not done: JoinWhere (non-equi join), common subexpression elimination, nested types (List/Struct/Map), and the long tail of Expr.Rolling*, Upsample, Interpolate and the trigonometric block.

Version numbers follow Go's own rule for v0: nothing is promised. The API is still moving, and the preamble above says why.

Correctness

make test-all   # four SIMD widths (512/256/128/0) plus the experiment off
make race       # the whole suite under -race
make levels     # import-level invariants

1386 test cases, and the matrix is not decoration. Vector width is a runtime property, so a single-width run proves very little: 512-bit gives 8 float64 lanes, which happens to be exactly one bitmap byte — a coincidence that hides an entire class of sub-byte bitmap bug. The 128-bit leg is where those surface.

Correctness is also checked against other engines. The benchmark suite validates every result against a duckdb reference, and ursus currently passes 22/22 PDS-H (TPC-H) queries and 15/15 h2o.ai queries.

Benchmarks

Everything that measures ursus lives in bench/ — three tiers behind one Makefile, competing against duckdb, polars, pandas, DataFusion, chDB, Arrow-Go, Gota and QFrame.

cd bench
make preflight    # refuses to run on a machine without the free memory; that is the feature
make setup
make bench        # gen -> reference answers -> run -> validate -> report

See bench/README.md for what is timed and why.

bench/results/REPORT.md is checked in. Every result in it is validated against a duckdb reference, and a disagreement is struck through rather than quietly reported as a fast number.

How current it is, precisely. The PDS-H table and h2o's gb7 row are freshly measured. The other fourteen h2o rows predate the parallel-aggregation, sort and min/max work, so the h2o geomean understates the engine — re-running that suite takes tens of minutes of full CPU and is not something to do casually. Timings come from a laptop under real conditions, so treat small differences as noise and the ordering as the signal.

ursus is not as fast as polars or duckdb. Where it is behind and why is recorded rather than glossed — see context_files/.

How it is built

dtype/          types, schemas, the null contract
i128/           128-bit integers, for Decimal
internal/
  data/         Column and Batch — Arrow layout, three payload shapes
  bitmap/       validity and boolean bitmaps, offset-correct by construction
  kernel/       the compute kernels, SIMD with scalar twins
  expr/         the expression IR
  plan/         logical plan, resolution, the optimizer rules
  physical/     operators, the sink/breaker protocol, spilling
  exec/         the driver
  source/       parquet, csv, memory
ursustest/      assertions for testing code that uses ursus

Packages are arranged in strict import levels — internal/gen/levels fails the build if a package imports one at or above its own level, which is what keeps the dependency graph a DAG rather than a suggestion.

Design notes

context_files/ holds the design documents and an as-built record for every step of construction, each written to be authoritative over the ones before it. They are unusually candid: they record the defects found, the measurements taken, the tests that turned out to be vacuous, and the claims that did not survive contact with the code. Start with the newest.

Licence

MIT.

Documentation

Overview

Package ursus is a Polars-class dataframe library for Go: an expression DSL, lazy execution with a query optimizer, Arrow memory, and SIMD kernels.

A tour 

df, err := ursus.Scan(src).
	Filter(ursus.Col("price").Gt(5)).
	Select(ursus.Col("id"), ursus.Col("price")).
	Collect(ctx)

Nothing runs until Collect. In between, ursus resolves the schema, expands any multi-column selections, type-checks every expression, and pushes the projection down into the scan so only the columns actually used are ever read.

The design in ten lines

  1. Lazy is the real API. Eager helpers are thin wrappers over it.
  2. Public types are concrete structs — Go forbids generic methods on interfaces, and generic methods are the point.
  3. Errors are deferred and sticky, surfaced at Collect, so chaining stays clean.
  4. context.Context appears at execution boundaries only, never in builders.
  5. Named methods, never operator tricks: a.Gt(5), not a > 5.
  6. Functional options where the parameter surface is wide.
  7. Generics at the boundary where Go values meet columns, not in the engine.
  8. Iterators for streaming output.
  9. No row index: row order is a property, not a label space.
  10. Null is not NaN, and both are handled explicitly everywhere.

Building

ursus requires Go 1.27 and GOEXPERIMENT=simd. Use the Makefile, or export the variable yourself — package simd does not compile without it.

Index

Constants

View Source 
const (
	// ClosedDefault is the zero value and means "this operation's own convention":
	// both for IsBetween, left for GroupByDynamic, right for Rolling.
	ClosedDefault = expr.ClosedDefault
	ClosedLeft    = expr.ClosedLeft
	ClosedRight   = expr.ClosedRight
	ClosedBoth    = expr.ClosedBoth
	ClosedNone    = expr.ClosedNone
)

The four window boundary conventions. ClosedLeft is the default for a window grid — an instant exactly on a boundary starts the new window — because it is the only one under which consecutive windows tile the line exactly.

View Source 
const (
	// InterpLinear interpolates proportionally between the neighbours. It is the
	// conventional default and the only option whose result need not be an input
	// value.
	InterpLinear = expr.InterpLinear

	// InterpLower and InterpHigher take the neighbour below or above the rank.
	InterpLower  = expr.InterpLower
	InterpHigher = expr.InterpHigher

	// InterpNearest takes the closer neighbour, rounding halves upward.
	InterpNearest = expr.InterpNearest

	// InterpMidpoint averages the two neighbours regardless of the rank's position
	// between them.
	InterpMidpoint = expr.InterpMidpoint
)
View Source 
const (
	// AsOfBackward is the default: the last right key at or before the left key.
	// "Attach the most recent quote to each trade."
	AsOfBackward = plan.AsOfBackward
	// AsOfForward is the first right key at or after the left key.
	AsOfForward = plan.AsOfForward
	// AsOfNearest is whichever is closer, ties going backward.
	AsOfNearest = plan.AsOfNearest
)
View Source 
const (
	// ConcatStrict requires the same columns in the same order — the default.
	//
	// Types still promote and nullability still widens. Neither is a relaxation:
	// refusing to stack two frames that differ only in a nullability flag would be
	// absurd, because that is the ordinary result of filtering one of them.
	ConcatStrict = plan.ConcatStrict

	// ConcatDiagonal takes the union of the columns, filling each frame's missing
	// ones with nulls. This is what makes stacking heterogeneous files work.
	ConcatDiagonal = plan.ConcatDiagonal
)
View Source 
const (
	// JoinInner keeps only matching pairs. The default.
	JoinInner = plan.JoinInner
	// JoinLeft keeps every left row, with right columns null where unmatched.
	JoinLeft = plan.JoinLeft
	// JoinRight is the mirror of JoinLeft.
	JoinRight = plan.JoinRight
	// JoinFull keeps every row from both sides.
	JoinFull = plan.JoinFull
	// JoinSemi keeps left rows that have at least one match. A filter: no right
	// column is added.
	JoinSemi = plan.JoinSemi
	// JoinAnti keeps left rows with no match. The inverse of JoinSemi.
	JoinAnti = plan.JoinAnti
	// JoinCross is the cartesian product. No keys.
	JoinCross = plan.JoinCross
)
View Source 
const (
	ValidateNone = plan.ValidateNone
	// ValidateOneToOne requires unique keys on both sides.
	ValidateOneToOne = plan.ValidateOneToOne
	// ValidateOneToMany requires unique keys on the LEFT.
	ValidateOneToMany = plan.ValidateOneToMany
	// ValidateManyToOne requires unique keys on the RIGHT. This is the one that
	// catches accidental fan-out, which is the most common analytics bug there is.
	ValidateManyToOne = plan.ValidateManyToOne
	// ValidateManyToMany imposes no constraint.
	ValidateManyToMany = plan.ValidateManyToMany
)
View Source 
const (
	Second = dtype.Second
	Milli  = dtype.Milli
	Micro  = dtype.Micro
	Nano   = dtype.Nano
)

Time resolutions.

View Source 
const (
	// MapGroupsToRows gives every row its own partition's value, in the original
	// row order. The default, and the only one that composes with other columns.
	MapGroupsToRows = expr.MapGroupsToRows

	// MapExplode leaves the output in partition order instead of permuting it back.
	// Cheaper, because it skips the inverse permutation — and it REORDERS the frame,
	// so it is only meaningful when every column is windowed the same way.
	MapExplode = expr.MapExplode

	// MapJoin would aggregate each partition into a List repeated across its rows.
	// Refused: ursus has no List column layout yet.
	MapJoin = expr.MapJoin
)
View Source 
const (
	// RankOrdinal gives every row a distinct rank, ties broken by input order.
	RankOrdinal = expr.RankOrdinal
	// RankDense gives tied rows the same rank and does not skip the next value:
	// [10,20,20,30] ranks as 1,2,2,3.
	RankDense = expr.RankDense
	// RankMin and RankMax give tied rows the lowest or highest of their positions:
	// [10,20,20,30] ranks as 1,2,2,4 and 1,3,3,4.
	RankMin = expr.RankMin
	RankMax = expr.RankMax
	// RankAverage gives tied rows the mean of their positions, so it returns
	// Float64 where every other method returns Uint32.
	RankAverage = expr.RankAverage
)

Variables

View Source 
var (
	Bool    = dtype.Bool
	Int8    = dtype.Int8
	Int16   = dtype.Int16
	Int32   = dtype.Int32
	Int64   = dtype.Int64
	Uint8   = dtype.Uint8
	Uint16  = dtype.Uint16
	Uint32  = dtype.Uint32
	Uint64  = dtype.Uint64
	Float32 = dtype.Float32
	Float64 = dtype.Float64
	// Int128 is the accumulator and output type of integer Sum. Widening to 128
	// bits is what makes an integer sum incapable of silently overflowing.
	Int128 = dtype.Int128
	String = dtype.String
	Binary = dtype.Binary
	Date   = dtype.Date
	NullT  = dtype.Null
)

Simple types.

View Source 
var (
	// ErrSchema is an unknown, duplicate or ambiguous column.
	ErrSchema = uerr.ErrSchema
	// ErrType is an operation not defined for the given types.
	ErrType = uerr.ErrType
	// ErrValue is a bad literal, an out-of-range cast, an unparseable pattern.
	ErrValue = uerr.ErrValue
	// ErrUnsupported is a well-formed request ursus cannot yet serve.
	ErrUnsupported = uerr.ErrUnsupported
	// ErrIO is a failure reading or writing a data source.
	ErrIO = uerr.ErrIO
	// ErrResource is a limit ursus refused to exceed: the query would have worked
	// with a larger WithMemoryLimit or a writable WithSpillDir.
	ErrResource = uerr.ErrResource
	// ErrInternal is a bug in ursus. Users should never legitimately see one.
	ErrInternal = uerr.ErrInternal
)

Error kinds, for branching on a failure without matching on its message. 

if errors.Is(err, ursus.ErrResource) { retry with a bigger memory limit }

These are re-exported because the sentinels themselves live under internal/, so no package outside this module could reach them — which made "a caller should be able to detect that without matching on a message" true only for callers inside the module.

View Source 
var (
	Datetime = dtype.Datetime
	Duration = dtype.Duration
	TimeOf   = dtype.Time
	Decimal  = dtype.Decimal
	List     = dtype.List
	Array    = dtype.Array
	Enum     = dtype.Enum
)

Parameterised type constructors.

View Source 
var (
	Of      = dtype.Of
	NotNull = dtype.NotNull
)

Of builds a nullable field; NotNull builds a non-nullable one.

Functions

This section is empty.

Types

type AsOfOption 

type AsOfOption func(*asOfCfg)

AsOfOption configures JoinAsOf.

func AsOfAllowExactMatches 

func AsOfAllowExactMatches(b bool) AsOfOption

AsOfAllowExactMatches decides whether a right key EQUAL to the left key may match. Default true.

func AsOfBy 

func AsOfBy(exprs ...Expr) AsOfOption

AsOfBy adds EXACT-match keys applied before the nearest-key search, so a trade matches only quotes for its own symbol.

Without it every left row searches one global run, which on any real fixture means matching the nearest row of the wrong instrument — a plausible number and the wrong one.

func AsOfLeftBy 

func AsOfLeftBy(exprs ...Expr) AsOfOption

AsOfLeftBy and AsOfRightBy are AsOfBy for differently-named columns.

func AsOfLeftOn 

func AsOfLeftOn(e Expr) AsOfOption

AsOfLeftOn and AsOfRightOn name differently-named ordering keys. Used together.

func AsOfOn 

func AsOfOn(e Expr) AsOfOption

AsOfOn names the ordering key, present under the same name on both sides.

func AsOfRightBy 

func AsOfRightBy(exprs ...Expr) AsOfOption

func AsOfRightOn 

func AsOfRightOn(e Expr) AsOfOption

func AsOfStrategyOpt 

func AsOfStrategyOpt(s AsOfStrategy) AsOfOption

AsOfStrategyOpt picks backward (the default), forward or nearest.

func AsOfSuffix 

func AsOfSuffix(s string) AsOfOption

AsOfSuffix renames a colliding right column. Defaults to Join's suffix.

func AsOfTolerance 

func AsOfTolerance(i Interval) AsOfOption

AsOfTolerance bounds how far the search may reach. A candidate further than this from the left key does not match, and the left row comes back null-padded.

The bound is recomputed per row for a CALENDAR interval, so Every("1mo") is a different distance in February than in March — which is the whole reason Interval is not a duration.

type AsOfStrategy 

type AsOfStrategy = plan.AsOfStrategy

AsOfStrategy picks which neighbouring right row a left row matches.

type CSVOption 

type CSVOption func(*csv.Options)

CSVOption configures ScanCSV.

func WithColumnNames 

func WithColumnNames(names ...string) CSVOption

WithColumnNames overrides the column names positionally.

func WithComment 

func WithComment(prefix string) CSVOption

WithComment makes lines starting with prefix comments. Multi-byte prefixes such as "//" are supported.

func WithHasHeader 

func WithHasHeader(b bool) CSVOption

WithHasHeader says whether the first record holds column names. Default true.

func WithInferRows 

func WithInferRows(n int) CSVOption

WithInferRows bounds how many records inference reads. 0 reads the whole file, which is exact and, on a large file, expensive. Default 100.

func WithMaxRecordSize 

func WithMaxRecordSize(n int) CSVOption

WithMaxRecordSize bounds a single record. Default 16 MiB. The bound exists so an unterminated quote is an error rather than an out-of-memory kill.

func WithNullValues 

func WithNullValues(vals ...string) CSVOption

WithNullValues adds texts that mean NULL. The empty string already means null for every type except String, where "" is a value.

func WithQuote 

func WithQuote(c byte) CSVOption

WithQuote sets the quote character. Default '"'.

func WithSchema 

func WithSchema(s *Schema) CSVOption

WithSchema supplies the schema and skips inference entirely. On a large file this is the option that matters: inference costs a read of the sample, and an explicit schema also removes the risk of a column being typed from rows that do not represent it.

func WithSchemaOverrides 

func WithSchemaOverrides(m map[string]dtype.DataType) CSVOption

WithSchemaOverrides fixes individual columns while inferring the rest. The usual case is an identity column that looks numeric and must not be.

func WithSeparator 

func WithSeparator(c byte) CSVOption

WithSeparator sets the field delimiter. Default ','.

func WithSkipRows 

func WithSkipRows(n int) CSVOption

WithSkipRows discards n records before the header.

func WithTruncateRaggedLines 

func WithTruncateRaggedLines(b bool) CSVOption

WithTruncateRaggedLines accepts records with the wrong field count: extra fields are dropped and missing ones become null. Off by default, because a changed field count usually means the file is not what the reader thinks it is.

type CSVSinkOption 

type CSVSinkOption interface {
	// contains filtered or unexported methods
}

CSVSinkOption is anything SinkCSV and WriteCSV accept: a writer option or an execution option. See ParquetSinkOption for why it is an interface.

type CSVWriteOption 

type CSVWriteOption func(*csv.WriteOptions)

CSVWriteOption configures the CSV writer.

func WithLineTerminator 

func WithLineTerminator(s string) CSVWriteOption

WithLineTerminator sets the record separator. Default "\n".

func WithNullValue 

func WithNullValue(s string) CSVWriteOption

WithNullValue sets the text written for a null. Default "".

The default reads back as null for every type except String, where "" is a real value the reader cannot distinguish from a missing one. A round trip that must preserve null strings needs a sentinel on both sides: WithNullValue("\\N") here and WithNullValues("\\N") on the read.

func WithWriteHeader 

func WithWriteHeader(b bool) CSVWriteOption

WithWriteHeader says whether to write column names first. Default true.

func WithWriteSeparator 

func WithWriteSeparator(c byte) CSVWriteOption

WithWriteSeparator sets the delimiter. Default ','.

type Closed 

type Closed = expr.Closed

Closed says which endpoints of an interval belong to it: [lo, hi), (lo, hi], [lo, hi] or (lo, hi). Used by IsBetween and by the temporal group-by's windows.

type CollectOption 

type CollectOption func(*collectCfg)

CollectOption configures execution.

func WithBatchSize 

func WithBatchSize(n int) CollectOption

WithBatchSize sets the rows per batch. Results must not depend on it; the test suite varies it precisely to check that.

func WithMemoryLimit 

func WithMemoryLimit(bytes int64) CollectOption

WithMemoryLimit caps what buffering operators may hold, in bytes. 

df, err := ursus.ScanParquet("120gb/*.parquet").
    Sort(ursus.Desc(ursus.Col("ts"))).
    SinkParquet(ctx, "sorted.parquet",
        ursus.WithMemoryLimit(512<<20),
        ursus.WithSpillDir("/tmp/ursus"))

What it bounds

Sort spills past the limit and merges the runs back, so a sort over more data than memory works. Group-by spills too, by radix-partitioning the keys it cannot hold: past the limit new keys are routed to one of sixteen files per level and re-aggregated afterwards. Join partitions BOTH sides on the join key and replays the pairs, so an equi-join over more data than memory works as well. Window, reverse, hstack, unique and tail cannot spill: past the limit they FAIL, with an error naming the operator, which is a better outcome than being killed by the OS with no explanation.

A bounded sort — one under a Head or a TopK — never spills at all: past the limit it discards everything outside the current best k, which is exact.

The one thing the limit CAN change

An unordered group-by's ROW ORDER depends on the limit, because the point at which it stops admitting new keys does. Past that point a new key goes to a partition file and comes back after every resident group, so the output is resident-first then partition-major. A spilling JOIN is the same, for the same reason: rows whose key stayed in memory come out in probe order, and the rest follow bucket by bucket.

The CONTENT is invariant: same rows, same values, at every limit and at none. Only the sequence moves. A group-by can opt out with GroupBy.MaintainOrder, which restores first-appearance order exactly, spilled or not.

A JOIN CANNOT, and that is a decision rather than an omission. Restoring probe order means holding the whole output to permute it, and a join's output can be larger than both of its inputs — so the flag would only work in the cases that did not need it. TestMemoryLimitIsNotASemanticKnob pins the content half for both, and TestJoinOrderIsUnspecifiedUnderALimit pins that the order really does move.

What a group-by's limit does NOT bound

Partitioning divides the KEY SPACE, so it bounds a group-by whose problem is cardinality. It cannot divide a single key, and quantile, median and n_unique hold state per VALUE rather than per group — so a group-by over a few very large groups still fails, with an error that says which of the two situations it is in.

The join has the same wall in two places: a CROSS join has no key at all, and one join key with more build rows than the limit cannot be split however deep the recursion goes. Both refuse with a message naming which it is.

What it does not bound

Collect materialises the whole result by definition, and asking for a frame in memory is a request to hold it. The larger-than-RAM story runs through SinkParquet and CollectBatches, both of which stream.

func WithMemoryStats 

func WithMemoryStats(out *MemoryStats) CollectOption

WithMemoryStats stores the query's memory accounting into *out when the query finishes.

It is written at the end of Collect, Count, CollectInto, SinkParquet, SinkCSV, WriteParquet and WriteCSV, and after the last batch of CollectBatches — so reading it early gives a partial figure rather than a wrong one.

The sinks matter most and were the ones this list used to omit: they are the consumers that stream, so they are where a memory limit is worth setting.

func WithOptFlags 

func WithOptFlags(f plan.Flags) CollectOption

WithOptFlags overrides which optimizer rules run. Use it to benchmark a rule's value or to bisect a wrong answer down to the rule that caused it.

func WithSpillDir 

func WithSpillDir(dir string) CollectOption

WithSpillDir chooses where spill files are written. The default is the system temporary directory. Files are removed when the query finishes, including on error and on an early break out of CollectBatches.

func WithThreads 

func WithThreads(n int) CollectOption

WithThreads sets how many workers a query runs on. Default runtime.NumCPU().

Results do not depend on it. Parallelism over the PIPELINE — the scan and the stateless per-batch operators above it — is ORDER-PRESERVING: workers process batches concurrently and the results are read back in input order, so the same query returns byte-identical output at any thread count. That is worth stating because it is not what every engine does, and because five things in ursus quietly depend on batch order — stable sort, First and Last, distinct's first-row rule, limit, and first-appearance group ids.

WithThreads(1) reproduces the serial operator tree exactly, which is the knob to reach for when bisecting.

Group-by also runs N-ways, and only where that cannot change an answer

Since step 17 a hash aggregation is drained by several sinks whose partial tables are merged. It is NOT order-preserving in the same structural way — the dispatcher is round-robin, so no worker holds a contiguous portion of the input — so it is used only where the result cannot depend on order at all: not under MaintainOrder, not with First, Last, ArgMin or ArgMax, and not under a memory limit, because a sink that has spilled cannot be merged. Every one of those falls back to the serial path silently and exactly.

Sort and join breakers remain serial. They still benefit, because their input arrives faster.

The trade, stated: a parallel group-by holds roughly N times the group table, because each worker builds its own. That is why the memory limit disables it rather than dividing the budget.

func WithVerify 

func WithVerify() CollectOption

WithVerify makes the optimizer assert schema preservation after every rule. On in tests, off by default because it costs a schema resolution per rule.

type Column 

type Column = data.Column

Column is a named, typed, nullable run of values.

func Values 

func Values[T Literal](name string, vals []T) *Column

Values builds a column from Go values, with no nulls.

The element type determines the column type: []int64 gives Int64, []string gives String, []time.Time gives Datetime(ns, UTC).

func ValuesNullable 

func ValuesNullable[T Literal](name string, vals []T, valid []bool) *Column

ValuesNullable builds a column with an explicit validity mask, where valid[i] false means row i is null.

Nulls are a parallel bitmap rather than a *T or an Option[T] because boxing would cost an allocation per row and make the values buffer non-contiguous — exactly what a columnar layout must not do.

type ConcatMode 

type ConcatMode = plan.ConcatMode

ConcatMode says how much Concat will reconcile differing schemas.

type ConcatOption 

type ConcatOption func(*concatCfg)

ConcatOption configures Concat.

func WithConcatMode 

func WithConcatMode(m ConcatMode) ConcatOption

WithConcatMode selects strict or diagonal reconciliation.

type DataFrame 

type DataFrame struct {
	// contains filtered or unexported fields
}

DataFrame is a materialised result: named, equal-length, typed columns.

There is no row index. Row order is a property of the data, not an addressable label space — the single most consequential thing Polars got right and pandas did not.

func (*DataFrame) At 

func (df *DataFrame) At[T any](row int, name string) (T, bool, error)

At returns one value and whether it is non-null.

func (*DataFrame) Batch 

func (df *DataFrame) Batch() *data.Batch

Batch exposes the underlying batch for internal use and tests.

func (*DataFrame) Column 

func (df *DataFrame) Column[T any](name string) (*data.Series[T], error)

Column returns a typed view of a column.

A generic METHOD — the thing Go 1.27 unlocked. Previously this had to be `ursus.Column[int64](df, "x")`, with the receiver demoted to an argument.

func (*DataFrame) Columns 

func (df *DataFrame) Columns() []string

Columns returns the column names in order.

func (*DataFrame) Concat 

func (df *DataFrame) Concat(ctx context.Context, others ...*DataFrame) (*DataFrame, error)

Concat stacks frames vertically.

func (*DataFrame) Drop 

func (df *DataFrame) Drop(ctx context.Context, names ...string) (*DataFrame, error)

Drop removes columns; Rename changes their names.

func (*DataFrame) Filter 

func (df *DataFrame) Filter(ctx context.Context, preds ...Expr) (*DataFrame, error)

Filter keeps rows where every predicate is true.

func (*DataFrame) Head 

func (df *DataFrame) Head(ctx context.Context, n int) (*DataFrame, error)

Head keeps the first n rows; Tail the last n.

func (*DataFrame) Height 

func (df *DataFrame) Height() int

Height returns the number of rows; Width the number of columns.

func (*DataFrame) Join 

func (df *DataFrame) Join(ctx context.Context, other *DataFrame, opts ...JoinOption) (*DataFrame, error)

Join combines two frames on keys.

func (*DataFrame) Lazy 

func (df *DataFrame) Lazy() *LazyFrame

Lazy turns a materialised frame back into a query.

The obvious spelling is wrong in three specific ways

`ursus.Frame(df.Batch().Columns()...)` compiles and runs, because Column is a type alias — and it is wrong:

  • It would re-derive nullability from the data. Field.Nullable is a static property of the schema, not a count of nulls actually present, so a nullable column that happens to hold none — the ordinary result of a filter — would come back non-nullable and every downstream join and cast would reason from the wrong schema.
  • It would lose the row count of a frame with rows but no columns.
  • It would alias the batch's own column slice, against Columns()' contract.

So this goes through the batch, which already carries the declared schema and the row count.

It is free: no copy, no re-derivation. The batch is immutable and shared.

func (*DataFrame) Rename 

func (df *DataFrame) Rename(ctx context.Context, names map[string]string) (*DataFrame, error)

func (*DataFrame) Reverse 

func (df *DataFrame) Reverse(ctx context.Context) (*DataFrame, error)

Reverse emits rows in the opposite order.

func (*DataFrame) Rows 

func (df *DataFrame) Rows[T any]() ([]T, error)

Rows decodes every row into a T.

func (*DataFrame) Schema 

func (df *DataFrame) Schema() *Schema

Schema returns the column names and types.

func (*DataFrame) Select 

func (df *DataFrame) Select(ctx context.Context, exprs ...Expr) (*DataFrame, error)

Select evaluates expressions and returns the result.

func (*DataFrame) Shape 

func (df *DataFrame) Shape() (int, int)

Shape returns (rows, columns).

func (*DataFrame) Sort 

func (df *DataFrame) Sort(ctx context.Context, keys ...SortKey) (*DataFrame, error)

Sort orders rows.

func (*DataFrame) String 

func (df *DataFrame) String() string

String renders the frame as an aligned table.

Display quality is not cosmetic: this is what people look at all day, and a column of unaligned numbers with no visible types is the difference between a library that feels finished and one that does not.

func (*DataFrame) Tail 

func (df *DataFrame) Tail(ctx context.Context, n int) (*DataFrame, error)

func (*DataFrame) Unique 

func (df *DataFrame) Unique(ctx context.Context, subset ...string) (*DataFrame, error)

Unique removes duplicate rows.

func (*DataFrame) Width 

func (df *DataFrame) Width() int

func (*DataFrame) WithColumns 

func (df *DataFrame) WithColumns(ctx context.Context, exprs ...Expr) (*DataFrame, error)

WithColumns adds or replaces columns.

type DataType 

type DataType = dtype.DataType

DataType is a logical column type. Comparable, usable as a map key.

func StructOf 

func StructOf(fields ...Field) DataType

StructOf builds a struct type. Named StructOf rather than Struct to leave the bare name available and to match DateOf / TimeOf.

type DtExpr 

type DtExpr struct {
	// contains filtered or unexported fields
}

DtExpr is the temporal namespace: `Col("ts").Dt().Year()`.

Components are read in the column's own timezone

2024-01-01T00:00:00+09:00 is hour 0 in Tokyo and hour 15 in UTC. A Datetime carries its zone, and every component below is read in it — otherwise `hour` would be wrong for every row of any non-UTC column, silently.

Durations have no components

`.Year()` on a Duration is refused at plan time. A duration is a length, not a point in time; "the year of 90 minutes" has no answer. Use the Total* family.

func (DtExpr) Day 

func (d DtExpr) Day() Expr

func (DtExpr) Epoch 

func (d DtExpr) Epoch() Expr

Epoch is whole seconds since 1970-01-01T00:00:00Z.

func (DtExpr) Hour 

func (d DtExpr) Hour() Expr

func (DtExpr) Microsecond 

func (d DtExpr) Microsecond() Expr

func (DtExpr) Millisecond 

func (d DtExpr) Millisecond() Expr

func (DtExpr) Minute 

func (d DtExpr) Minute() Expr

func (DtExpr) Month 

func (d DtExpr) Month() Expr

func (DtExpr) Nanosecond 

func (d DtExpr) Nanosecond() Expr

func (DtExpr) OrdinalDay 

func (d DtExpr) OrdinalDay() Expr

OrdinalDay is the day of the year, 1-366.

func (DtExpr) Quarter 

func (d DtExpr) Quarter() Expr

Quarter is 1-4.

func (DtExpr) Second 

func (d DtExpr) Second() Expr

func (DtExpr) ToString 

func (d DtExpr) ToString() Expr

ToString formats using the same ISO 8601 layouts the CSV writer emits, so a frame printed to a terminal and a frame written to a file agree.

func (DtExpr) TotalDays 

func (d DtExpr) TotalDays() Expr

The Total family converts a Duration to a whole number of units, truncating. They are the only temporal functions defined on a Duration.

func (DtExpr) TotalHours 

func (d DtExpr) TotalHours() Expr

func (DtExpr) TotalMinutes 

func (d DtExpr) TotalMinutes() Expr

func (DtExpr) TotalSeconds 

func (d DtExpr) TotalSeconds() Expr

func (DtExpr) Truncate 

func (d DtExpr) Truncate[T Span](every T) Expr

Truncate floors an instant to a multiple of every, toward negative infinity so instants before the epoch move backwards like every other instant.

A Duration floors ABSOLUTELY; an Interval floors on the CALENDAR

The two spellings answer different questions and neither is a special case of the other: 

Truncate(time.Hour)     the instant, floored to a whole hour since the epoch
Truncate(Every("1d"))   the start of the LOCAL day, in the column's own zone

A Duration is a fixed span of elapsed time, so flooring by one is zone-independent by definition — `Truncate(24*time.Hour)` on a New York column lands on 00:00 UTC, which is 19:00 or 20:00 local, and that is the correct answer to the question a Duration asks. It is almost never the question a user grouping by day is asking, which is why Every("1d") exists and reads the column's timezone.

func (DtExpr) Week 

func (d DtExpr) Week() Expr

Week is the ISO 8601 week number, which is not "day of year / 7" — the first week of a year can begin in the previous one.

func (DtExpr) Weekday 

func (d DtExpr) Weekday() Expr

Weekday is ISO: Monday is 1 and Sunday is 7. Go's time.Weekday puts Sunday at 0, and passing that through is the classic off-by-one in date libraries.

func (DtExpr) Year 

func (d DtExpr) Year() Expr

type DynamicOptions 

type DynamicOptions struct {
	// Every is where windows START. Consecutive windows are Every apart.
	Every Interval

	// Period is a window's WIDTH. Zero means Every, which gives windows that tile
	// the input exactly. A Period larger than Every gives OVERLAPPING windows, and a
	// row then belongs to several of them — so a Len() over the result exceeds the
	// input's row count, which is the correct answer and surprises everyone once.
	Period Interval

	// Offset shifts the whole grid. A day grid with Offset 9h starts its windows at
	// 09:00 rather than at midnight.
	Offset Interval

	// Closed says which end of a window belongs to it. The zero value is
	// ClosedLeft — [start, end) — which is the only convention under which a grid
	// partitions its input exactly.
	Closed Closed

	// GroupBy adds CATEGORICAL keys. Windows are cut independently inside each
	// distinct combination, so an hourly rollup per service is one call.
	GroupBy []Expr

	// Label, StartBy and IncludeBoundaries are not implemented. They are declared so
	// that setting one is an error naming it rather than silence.
	Label             string
	StartBy           string
	IncludeBoundaries bool
}

DynamicOptions configures GroupByDynamic.

Five fields, and three that are refused rather than ignored

Every, Period, Offset, Closed and GroupBy are implemented. Label, StartBy and IncludeBoundaries are named in the design docs and are NOT — and they produce an error saying so rather than being accepted and dropped.

That is deliberate and it is a lesson this library has already paid for once: plan.Aggregate.MaintainOrder sat assigned, rendered and unread for four steps, and its own doc had to say "the flag documents the guarantee rather than changing behaviour". A field that is accepted and does nothing is worse than one that does not exist, because the next reader believes it.

type ExplainOption 

type ExplainOption func(*explainCfg)

ExplainOption configures Explain.

func Optimized 

func Optimized(b bool) ExplainOption

Optimized selects the optimized plan (the default) or the plan as written.

func WithSchemas 

func WithSchemas() ExplainOption

WithSchemas annotates every plan node with its output schema.

type Expr 

type Expr struct {
	// contains filtered or unexported fields
}

Expr is a lazy description of a column transformation.

It is a CONCRETE STRUCT rather than an interface, and that is forced by Go 1.27: interface methods may not declare type parameters, and a generic method can never satisfy an interface method. Since Gt[T Operand] — the thing that makes `Col("age").Gt(30)` work without wrapping 30 in Lit — must be a generic method, Expr cannot be an interface. Polymorphism lives in the unexported expr.Node instead.

func All 

func All() Expr

All selects every column.

func Coalesce 

func Coalesce(exprs ...Expr) Expr

Coalesce returns the first non-null value across the given expressions.

Built out of conditionals rather than its own kernel

Coalesce(a, b, c) becomes when(a.is_not_null()).then(a).otherwise(coalesce(b, c)), which is correct and costs one thing worth knowing: each operand is MENTIONED twice, and there is no common-subexpression pass, so each is EVALUATED twice. For the ordinary Coalesce(Col("a"), Col("b")) that is two extra column reads; for an operand that is itself an expensive expression it is not free.

A dedicated n-ary null-merge kernel would remove that and change no semantics. It is deliberately not built yet: the desugaring needs no new IR node, no new kernel and no new arms in the five type switches a node type must reach.

func Col 

func Col(names ...string) Expr

Col selects one or more columns by name.

With several names it becomes a multi-column selection that expands at plan time: `Col("a","b").Mul(2)` is two independent expressions.

func ColDType 

func ColDType(types ...dtype.DataType) Expr

ColDType selects every column of one of the given types. How many columns that is is determined at plan time from the schema, so the same expression adapts as the schema changes.

func ColRegex 

func ColRegex(pattern string) Expr

ColRegex selects every column whose name matches pattern.

The pattern is explicit rather than inferred from leading ^ and trailing $ the way Polars does. That inference makes a column literally named "^total$" unselectable and taxes every plain Col call with an anchor scan.

func Exclude 

func Exclude(names ...string) Expr

Exclude selects every column except the named ones.

func ExcludeRegex 

func ExcludeRegex(pattern string) Expr

ExcludeRegex selects every column whose name does not match pattern.

func Len 

func Len() Expr

Len counts rows in the current group — SQL's COUNT(*) with no column argument.

It aggregates over a literal rather than a column, because counting rows must not depend on any column existing: `GroupBy(k).Agg(Len())` has to work on a frame whose only column is the key.

func Lit 

func Lit[T Literal](v T) Expr

Lit builds a literal.

func Null 

func Null(dt dtype.DataType) Expr

Null builds a typed null literal.

func (Expr) Abs 

func (e Expr) Abs() Expr

func (Expr) Add 

func (e Expr) Add[T Operand](v T) Expr

func (Expr) Alias 

func (e Expr) Alias(name string) Expr

Alias renames the output column.

It sets ONE fixed name, so applying it to a multi-column selection is an error. Use Prefix or Suffix to rename an expansion.

func (Expr) AllTrue 

func (e Expr) AllTrue() Expr

func (Expr) And 

func (e Expr) And[T Operand](v T) Expr

func (Expr) Any 

func (e Expr) Any() Expr

Any and AllTrue reduce a Boolean column.

AllTrue rather than All, because All is already the top-level selector for every column and one name cannot mean both.

Nulls are SKIPPED, so a group with no non-null value gives NULL rather than false (for Any) or true (for AllTrue). Polars returns the vacuous answer; ursus returns null, because every other aggregate here does and an inconsistent one is a permanent trap. FillNull makes either convention available.

func (Expr) ArgMax 

func (e Expr) ArgMax() Expr

func (Expr) ArgMin 

func (e Expr) ArgMin() Expr

ArgMin and ArgMax give the POSITION of the smallest or largest value within the group, as a Uint64.

The index counts every row of the group, nulls included, so it lines up with anything else gathered from the same group — but nulls can never win, and a group with no non-null value has no position and returns NULL.

Ties go to the earliest row, which is what makes the answer independent of batch size and thread count.

func (Expr) BackwardFill 

func (e Expr) BackwardFill(limit int) Expr

BackwardFill replaces each null with the next non-null value after it.

It is ForwardFill walked in the other direction, which costs nothing: the ordered walk flips its direction rather than building a second permutation.

func (Expr) Cast 

func (e Expr) Cast(to dtype.DataType) Expr

Cast converts to another type, failing the query on the first unrepresentable value.

func (Expr) CastLossy 

func (e Expr) CastLossy(to dtype.DataType) Expr

CastLossy converts to another type, turning unrepresentable values into nulls. The result is always nullable as a consequence.

func (Expr) Cbrt 

func (e Expr) Cbrt() Expr

func (Expr) Ceil 

func (e Expr) Ceil() Expr

func (Expr) Clip 

func (e Expr) Clip[L, H Operand](lo L, hi H) Expr

Clip bounds each value to [lo, hi].

Sugar, and that is what lets the bounds be expressions 

When(e.Lt(lo)).Then(lo).When(e.Gt(hi)).Then(hi).Otherwise(e)

A parameterised Call could not do this: CallArgs requires every argument to be a literal and rejects a column-valued one by name. As a conditional, Clip takes Expr bounds naturally — Col("x").Clip(Col("floor"), Col("cap")) works.

The bounds are WEAK literals, like a fill value

So Col("i32").Clip(0, 100) stays Int32. It used to become Int64, because the When/Then builder lifts strongly — and Col("u64").Clip(0, 100) became Int128, which is a startling type for clipping an unsigned column to a small range. That is exactly the inconsistency weak literals were built to remove: both spellings look like "a Go scalar bound on a column", and only one of them preserved the type. A bound that does not fit falls back to ordinary promotion, the same as a fill value that does not fit.

An Expr bound is used as written, so Clip(Lit(0), Lit(100)) still widens.

Edge cases, each of which falls out rather than being coded 

e is null      → the comparisons are null → neither branch → null
e is NaN       → NaN < lo and NaN > hi are both false → e, unchanged
a bound is null → every row's mask is null → the whole column is null
lo > hi        → lo wins, because the chain is right-nested in written order

The third is the one to watch: a null bound is not "no bound", it is a null result. Use two separate clips if only one side should apply.

func (Expr) Count 

func (e Expr) Count() Expr

Count counts NON-NULL values — SQL's COUNT(col).

func (Expr) CumCount 

func (e Expr) CumCount(reverse bool) Expr

CumCount numbers rows within the window, starting at 1.

With reverse it counts from the end, so the last row is 1. That is what makes IsLastDistinct expressible as a window rather than a second pass.

func (Expr) CumMax 

func (e Expr) CumMax(reverse bool) Expr

func (Expr) CumMin 

func (e Expr) CumMin(reverse bool) Expr

func (Expr) CumProd 

func (e Expr) CumProd(reverse bool) Expr

func (Expr) CumSum 

func (e Expr) CumSum(reverse bool) Expr

CumSum, CumProd, CumMin and CumMax are running reductions over the window.

Nulls are SKIPPED by the running value and PRESERVED in place: cum_sum of [1, null, 3] is [1, null, 4], not [1, 1, 4] and not [1, null, null]. The running total ignores what is not there; the output still says the value was missing.

CumSum uses the same accumulator width as Sum, so the last row of `cum_sum(x)` equals `sum(x)` rather than differing by an overflow. CumProd returns Float64 for the reason Product does: i128 has addition but no multiplication.

func (Expr) Diff 

func (e Expr) Diff(n int) Expr

Diff is the difference from the value n rows earlier.

It costs ONE shift, not two

`e.Sub(e.Shift(n))` mentions the shift twice, and window resolution dedups on the rendered expression — so both mentions resolve to the same temporary and the shift is computed once. That is real common-subexpression elimination, unlike Coalesce, which has none.

The type is whatever subtraction gives, which is the operand's own for a number and a DURATION for an instant — the temporal algebra's instant-minus-instant rule. Polars' diff carries a null_behavior parameter this composition cannot express; the first n rows are null here.

func (Expr) Div 

func (e Expr) Div[T Operand](v T) Expr

Div is TRUE division: it always produces a float, so 7/2 is 3.5 even for two integers. Integer-truncating division is FloorDiv, so the surprising behaviour has to be asked for by name.

func (Expr) DropNans 

func (e Expr) DropNans() Expr

DropNans is refused for the same reason as DropNulls, and its hint carries the asymmetry that keeps a frame-level DropNans from being written the obvious way.

func (Expr) DropNulls 

func (e Expr) DropNulls() Expr

DropNulls is not an expression. It is refused here so the mistake is caught at build time with a message naming the operation that does exist.

Why there is no expression form

Every expression must produce exactly one value per input row — the evaluator enforces it — and dropping rows changes the frame's height. There is no Expr.Filter, Expr.Slice or Expr.Gather family to hang a height-changing expression on, so this is not a missing case but a different shape of thing.

Refused at BUILD time rather than at plan time on purpose: the deferred error is the first thing expression resolution looks at, so CollectSchema and Explain both fail too. A refusal that only Collect notices — which is what happened to one of the window mapping strategies — lets Explain print a plan that cannot run.

func (Expr) Dt 

func (e Expr) Dt() DtExpr

Dt opens the temporal namespace.

func (Expr) Eq 

func (e Expr) Eq[T Operand](v T) Expr

func (Expr) EqMissing 

func (e Expr) EqMissing[T Operand](v T) Expr

EqMissing is equality where nulls are DATA: null equals null, and the result is never null. NeMissing is its negation.

func (Expr) Err 

func (e Expr) Err() error

Err returns any error deferred during construction — a bad regex, say. Expression builders cannot return errors without destroying chaining, so the error rides in the tree and surfaces here or at Collect.

func (Expr) Exp 

func (e Expr) Exp() Expr

func (Expr) FillNan 

func (e Expr) FillNan[T Operand](v T) Expr

FillNan replaces NaN with a value, leaving nulls alone.

Nulls survive, and the reason is worth knowing

`e.IsNotNan()` on a null row is NULL, not true — null and NaN are different things all the way down — and kernel.Select makes a null mask take NEITHER branch. So a null row keeps its null and the fill value never lands on it. That is exactly right, and it looks accidental, which is why TestFillNanLeavesNullsAlone exists.

Why the predicate is IsNotNan rather than IsNan

Both spell the same thing, and only one names the result correctly. OutputName takes the LEFTMOST column of an expression, so with `IsNan → Then: value, Else: e` the leftmost column reference is the literal and every filled column came back called "literal" — which for the frame-level form meant a new column instead of a replaced one.

func (Expr) FillNull 

func (e Expr) FillNull(s FillStrategy) Expr

FillNull replaces nulls according to a named strategy. 

Col("temp").FillNull(ursus.FillForward)
Col("qty").FillNull(ursus.FillZero)

For a forward or backward fill with a limit, use ForwardFill(n) or BackwardFill(n) directly — the strategy form is unlimited.

Three of these buffer the whole frame

FillMin, FillMax and FillMean compute an aggregate over every row before they can fill anything, so they turn a streaming query into one that holds its input. FillForward and FillBackward are ordered windows and buffer for the same reason. Only FillZero and FillOne stream.

func (Expr) FillNullWith 

func (e Expr) FillNullWith[T Operand](v T) Expr

FillNullWith replaces nulls with a value.

It does not widen the column 

Col("i32").FillNullWith(0)     // stays Int32
Coalesce(Col("i32"), Lit(0))   // becomes Int64

A Go scalar lifts to a WEAK literal, which adopts the column's own type when the value fits it exactly. That is deliberately not what arithmetic does: Col("i32").Add(1) still widens, because addition can overflow and filling a null cannot. An Expr operand is used as written, so FillNullWith(Lit(0)) widens.

If the value does not fit — FillNullWith(int64(5000)) on an Int8 column — the type falls back to ordinary promotion rather than truncating, so the result is an Int64 column holding 5000 rather than an Int8 column holding -120.

Cost

This desugars to a conditional, so the receiver is MENTIONED twice and, with no common-subexpression pass, EVALUATED twice — exactly as Coalesce is.

func (Expr) First 

func (e Expr) First() Expr

First and Last are POSITIONAL and may return null.

They are a selection rather than a reduction: skipping nulls would mean First(a) and First(b) could come from different rows, which is exactly what people use them together for.

func (Expr) Floor 

func (e Expr) Floor() Expr

Floor and Ceil round towards -Inf and +Inf. Both are the identity on an integer column and both are refused for Decimal, whose stored value is unscaled — so flooring it would floor 1234 rather than 12.34.

func (Expr) FloorDiv 

func (e Expr) FloorDiv[T Operand](v T) Expr

func (Expr) ForwardFill 

func (e Expr) ForwardFill(limit int) Expr

ForwardFill replaces each null with the last non-null value before it.

limit caps how many consecutive nulls one value may fill; 0 means unlimited. A leading run of nulls has nothing to carry and stays null.

Like every ordered function it is a window: with no .Over() it runs over the whole frame in scan order, which makes it a pipeline breaker.

func (Expr) Ge 

func (e Expr) Ge[T Operand](v T) Expr

func (Expr) Gt 

func (e Expr) Gt[T Operand](v T) Expr

func (Expr) IsBetween 

func (e Expr) IsBetween[L, H Operand](lo L, hi H, closed ...Closed) Expr

IsBetween tests that e falls between lo and hi. Both bounds lift, so IsBetween(0, 100) works.

The default is CLOSED-BOTH — lo <= e <= hi — which is what this method has always meant and what reads naturally from the name. Pass a Closed to say otherwise: 

Col("ts").IsBetween(start, end, ursus.ClosedLeft)   // start <= ts < end

Variadic rather than a required third argument, so every existing two-argument call keeps compiling and the common case stays short. More than one is a mistake and is refused rather than ignored.

func (Expr) IsClose 

func (e Expr) IsClose(other Expr, relTol, absTol float64) Expr

IsClose reports whether each value is within a relative or absolute tolerance of other. 

|a - b| <= max(relTol * max(|a|, |b|), absTol)

Both guards are load-bearing, and neither is obvious

The EQUALITY disjunct is what makes IsClose(Inf, Inf) true: the tolerance term alone would compute |Inf - Inf|, which is NaN, and NaN <= x is false.

The FINITENESS conjunct is what makes IsClose(Inf, -Inf) false. Without it the relative tolerance scales by max(|a|, |b|) = Inf, the limit becomes Inf, and Inf <= Inf says "close" — which it plainly is not. Python's math.isclose has the same special case for the same reason.

Together they give: equal infinities are close, opposite ones are not, and IsClose(NaN, NaN) is false because IEEE says NaN equals nothing including itself. With a null operand every term is null and Kleene OR gives null, matching every other comparison.

It computes in Float64

Both operands are cast first, so an integer column works — integers are always finite — and the finiteness test, which is float-only, has something to test.

Cost

The receiver is mentioned five times and other four, with no CSE. For two bare columns that is nine column reads; for expensive operands it is not free.

func (Expr) IsDuplicated 

func (e Expr) IsDuplicated() Expr

func (Expr) IsFinite 

func (e Expr) IsFinite() Expr

func (Expr) IsFirstDistinct 

func (e Expr) IsFirstDistinct() Expr

IsFirstDistinct and IsLastDistinct mark the first and last occurrence of each value, in input order.

Note the asymmetry with IsUnique: a value occurring three times has one first occurrence, one last, and is not unique anywhere.

func (Expr) IsIn 

func (e Expr) IsIn[T Literal](vs ...T) Expr

IsIn tests membership in a set of values: Col("region").IsIn("eu", "us").

Values, not an expression

The published sketch was IsIn(other Expr), and it cannot be written: Literal's only slice term is ~[]byte, so there is no way to spell a list-valued literal. The Expr-valued form — testing membership in another frame's column — is a semi join wearing a predicate's clothes, and it belongs with the join machinery.

All values share one type parameter, so the set is homogeneous by construction. Values that are not representable in the column's type are an error rather than a silent non-match: comparing an Int8 column against 5000 is a question with no meaningful answer.

Equality is GROUPING equality

Membership uses the same encoding GroupBy and Distinct use, so NaN matches NaN and -0.0 matches +0.0. An is_in built on IEEE equality would disagree with Distinct about the same data.

Nulls

`null.IsIn(...)` is NULL, not false — the same rule as IsNan, and the reason Filter(p) and Filter(Not(p)) do not partition. The set itself can never contain a null, because these are Go values.

An empty set makes every non-null row false, matching SQL's `x IN ()`.

func (Expr) IsInfinite 

func (e Expr) IsInfinite() Expr

func (Expr) IsLastDistinct 

func (e Expr) IsLastDistinct() Expr

func (Expr) IsNan 

func (e Expr) IsNan() Expr

IsNan and friends ask about a float VALUE, so `null.IsNan()` is null rather than false. Null and NaN are different things; use IsNull to test for missing.

func (Expr) IsNotNan 

func (e Expr) IsNotNan() Expr

func (Expr) IsNotNull 

func (e Expr) IsNotNull() Expr

func (Expr) IsNull 

func (e Expr) IsNull() Expr

IsNull and IsNotNull ask about presence, so their results are never null.

func (Expr) IsUnique 

func (e Expr) IsUnique() Expr

IsUnique reports whether each row's value occurs exactly once in the column. IsDuplicated is its negation for non-null rows.

These are windows, and that is why they were not built earlier

The answer for row 0 depends on the last row, so neither is elementwise. Step 7 deferred them precisely here, and they need no new machinery: a value is unique iff the partition keyed by that value has exactly one row.

Equality is GROUPING equality, the same the partitioning uses — so NaN equals NaN and -0.0 equals +0.0, and IsUnique agrees with Unique() about the same data. Nulls likewise group together, so two nulls are duplicates of each other.

func (Expr) Last 

func (e Expr) Last() Expr

func (Expr) Le 

func (e Expr) Le[T Operand](v T) Expr

func (Expr) Len 

func (e Expr) Len() Expr

Len counts ROWS, including nulls — SQL's COUNT(*).

Count and Len are deliberately separate and are never aliased: over a column with nulls they give different answers, and which one was meant is not recoverable after the fact.

func (Expr) Ln 

func (e Expr) Ln() Expr

Ln is the natural logarithm; Log10 is base 10; Log1p is ln(1+x), which keeps its precision for x near zero where ln(1+x) would lose it.

ln(0) is -Inf and ln(-1) is NaN. Both are values.

func (Expr) Log 

func (e Expr) Log(base float64) Expr

Log is the logarithm to an arbitrary base.

It is SUGAR — ln(x)/ln(base) — rather than a parameterised kernel, which is the same definition Polars uses and costs no new machinery. Two consequences worth knowing, both from the division:

  • Log(10) and Log10() can differ in the last ULP. Log10 is the exact form.
  • On a Float32 column Log10() stays Float32 and Log(10) becomes Float64, because dividing by a float64 promotes. Cast the result back if the narrow width was a deliberate choice.

func (Expr) Log1p 

func (e Expr) Log1p() Expr

func (Expr) Log10 

func (e Expr) Log10() Expr

func (Expr) Lt 

func (e Expr) Lt[T Operand](v T) Expr

func (Expr) MapName 

func (e Expr) MapName(fn func(string) string) Expr

MapName derives the output name from the input name.

func (Expr) Max 

func (e Expr) Max() Expr

func (Expr) Mean 

func (e Expr) Mean() Expr

Mean is the sum of non-null values divided by the COUNT of non-null values — never by the group size, which would silently bias the result downward whenever nulls are present.

Mean of a group with no non-null values is NULL, not NaN: there was nothing to compute, as opposed to a computation that came out undefined.

func (Expr) Median 

func (e Expr) Median() Expr

Median is the 0.5 quantile with linear interpolation, so an even-sized group gives the mean of the two middle values.

It returns Float64 for every numeric input, and skips nulls.

func (Expr) Min 

func (e Expr) Min() Expr

Min and Max use ursus's TOTAL order, the same one Sort uses: NaN sorts above everything and -0.0 equals +0.0. They therefore always agree with Sort(...).First() / .Last(), which IEEE comparison would not.

func (Expr) Mod 

func (e Expr) Mod[T Operand](v T) Expr

func (Expr) Mul 

func (e Expr) Mul[T Operand](v T) Expr

func (Expr) NUnique 

func (e Expr) NUnique() Expr

NUnique counts distinct non-null values.

Nulls are SKIPPED, so NUnique over an all-null group is 0. Polars counts null as a distinct value; ursus follows SQL and, more importantly, follows its own other aggregates.

func (Expr) Ne 

func (e Expr) Ne[T Operand](v T) Expr

func (Expr) NeMissing 

func (e Expr) NeMissing[T Operand](v T) Expr

func (Expr) Neg 

func (e Expr) Neg() Expr

Neg negates. Abs takes the absolute value.

Both are defined for every numeric type, including the unsigned integers, Int128 and Decimal — which matters more than it sounds, because every integer Sum outputs Int128, so `Col("x").Sum().Abs()` is an ordinary query.

func (Expr) Not 

func (e Expr) Not() Expr

Not is Kleene negation: NOT null is null.

func (Expr) NullCount 

func (e Expr) NullCount() Expr

NullCount counts the NULLS in each group — the complement of Count, and like the other counters it is never itself null.

func (Expr) Or 

func (e Expr) Or[T Operand](v T) Expr

func (Expr) Over 

func (e Expr) Over(partitionBy ...Expr) Expr

Over computes the expression within partitions and writes the answer back onto every row. 

// each row's deviation from its group's mean
ursus.Col("x").Sub(ursus.Col("x").Mean().Over(ursus.Col("g")))

// dense rank of speed within each type
ursus.Col("speed").Rank(ursus.RankDense, true).Over(ursus.Col("type"))

The distinction from GroupBy

GroupBy REDUCES: N rows in, one row per group out. Over PRESERVES: N rows in, N rows out, each carrying its own partition's answer. That is why an aggregate is refused in Select but the same aggregate under Over is not.

With no partition keys the whole frame is one partition, so `Col("x").Sum().Over()` puts the grand total on every row.

Nulls form their own partition

A null partition key groups with other nulls, the same rule GroupBy uses — as opposed to joins, where null keys match nothing. The two differ on purpose and each is documented where it applies.

func (Expr) OverWith 

func (e Expr) OverWith(spec WindowSpec) Expr

OverWith is Over with an ordering and a mapping strategy.

An ordering is what makes Rank, the cumulative functions and Shift meaningful: without one they run in input order, which is well-defined but rarely what a ranking wants.

func (Expr) PctChange 

func (e Expr) PctChange(n int) Expr

PctChange is the fractional change from the value n rows earlier.

Always a float on a numeric column, because it divides — and a zero previous value gives ±Inf rather than a null, since float division is total. On a temporal column it is a plan-time error: the difference is a Duration and the temporal algebra defines no Duration ÷ instant.

func (Expr) Pow 

func (e Expr) Pow[T Operand](v T) Expr

Pow raises the receiver to the power v.

The result is ALWAYS a float, for the same reason Div's is: 2**-1 is 0.5, and a version that truncated negative powers to zero would be surprising in the direction that loses data silently. Cast the result if an integer is wanted.

func (Expr) Prefix 

func (e Expr) Prefix(p string) Expr

Prefix prepends to the output name. Unlike Alias it is expansion-safe, deriving a distinct name per output column.

func (Expr) Product 

func (e Expr) Product() Expr

Product multiplies the non-null values.

Unlike Sum it returns FLOAT64 for every numeric input, including integers. Sum widened to Int128 so that overflow became unreachable; product cannot, because i128 has no multiplication — and an Int64 product wraps silently after roughly twenty ordinary factors. Float64 is exact to 2^53 and approximate above it, which is the weaker guarantee, stated rather than hidden.

Product of a group with no non-null values is NULL, not 1, following Sum.

func (Expr) Quantile 

func (e Expr) Quantile(q float64, interp Interpolation) Expr

Quantile returns the value at rank q, where q runs from 0 (the minimum) to 1 (the maximum). interp decides what happens when the rank falls between two values.

The rank is q·(n−1) over the non-null values — the convention numpy, Polars and DuckDB share — and the result is Float64 even for InterpLower and InterpHigher, which do return an actual input value.

Values are ordered by ursus's TOTAL order, so NaN sorts above everything and a high quantile of a column containing NaN is NaN, exactly as Sort would place it.

func (Expr) Rank 

func (e Expr) Rank(method RankMethod, descending bool) Expr

Rank numbers rows by value within the window, smallest first unless descending.

Ties are resolved by method; see RankMethod. Rank is never null — a null value still occupies a position — and returns Uint32, or Float64 for RankAverage.

func (Expr) Round 

func (e Expr) Round(decimals int) Expr

Round rounds to decimals decimal places, HALF AWAY FROM ZERO.

Round(0.5) is 1 and Round(2.5) is 3, matching Polars and a spreadsheet rather than IEEE's round-half-to-even. It preserves the operand's type: rounding an integer column is the identity, not a widening.

It rounds twice — the value is scaled by 10^decimals first, and that multiply rounds too. Round(2.675, 2) is 2.68, because 2.675 * 100 lands on exactly 267.5 even though 2.675 itself is stored as 2.67499999999999982…

func (Expr) Shift 

func (e Expr) Shift(n int) Expr

Shift moves values n places later within the window, filling the vacated rows with null. A negative n shifts earlier — Polars' lag and lead, one function.

Rows shifted in from outside the partition are null, never a neighbouring partition's value.

func (Expr) ShiftFill 

func (e Expr) ShiftFill[T Operand](n int, fill T) Expr

ShiftFill is Shift with a value instead of null in the vacated rows.

func (Expr) Sign 

func (e Expr) Sign() Expr

Sign is -1, 0 or 1, keeping the operand's type.

For floats it returns the operand itself in the zero case, so sign(-0.0) is -0.0 and sign(NaN) is NaN. That matches NumPy and falls out of the ordering test rather than being special-cased.

func (Expr) Sqrt 

func (e Expr) Sqrt() Expr

Sqrt, Cbrt and Exp. Sqrt of a negative is NaN, not an error and not a null.

func (Expr) Std 

func (e Expr) Std(ddof int) Expr

func (Expr) Str 

func (e Expr) Str() StrExpr

Str opens the string namespace.

func (Expr) String 

func (e Expr) String() string

String renders the expression canonically.

func (Expr) Sub 

func (e Expr) Sub[T Operand](v T) Expr

func (Expr) Suffix 

func (e Expr) Suffix(s string) Expr

Suffix appends to the output name. Expansion-safe, like Prefix.

func (Expr) Sum 

func (e Expr) Sum() Expr

Sum adds the non-null values.

An INTEGER sum accumulates and returns Int128, following DuckDB. Summing a thousand Int8 values overflows an Int8 immediately, and a wrapped sum is a plausible-looking wrong number that nothing downstream can detect. At 128 bits overflow needs 2^63 maximum-magnitude rows, i.e. it cannot happen.

A FLOAT sum accumulates in Float64 even when it returns Float32, because naive float32 accumulation stops making progress past ~2^24 elements.

Sum of a group with no non-null values is NULL, not 0 — ursus follows SQL here rather than Polars. `0` cannot be distinguished afterwards from a genuine zero; NULL can be turned into 0 with FillNull if that is what you want.

func (Expr) Var 

func (e Expr) Var(ddof int) Expr

Var and Std are the variance and standard deviation of the non-null values.

ddof is the delta degrees of freedom: 0 for the population statistic, 1 for the sample one. It has no default because there is no defensible default — Polars and pandas use 1, numpy uses 0, and silently picking either produces a number that is wrong for half its readers.

A group with ddof or fewer values returns NULL: the sample variance of a single observation is undefined, and 0 would claim the data has no spread rather than that the question has no answer.

Computed with Welford's algorithm, so a column of large near-equal values gives the right answer rather than the zero (or negative) that E[x²]−E[x]² produces.

func (Expr) Xor 

func (e Expr) Xor[T Operand](v T) Expr

type Field 

type Field = dtype.Field

Field is a named, typed, nullable column slot.

type FillStrategy 

type FillStrategy uint8

FillStrategy names a fill rule that needs no value at the call site.

It is a root-package enum and never reaches the IR, so it is not a dedup key and carries none of the collision hazard the expression enums do. 

const (
	// FillZero and FillOne fill with a constant. The literal is WEAK, so the
	// column's type is preserved rather than widened — which matters more here than
	// anywhere else, because the call site contains no literal at all to explain a
	// widening.
	FillZero FillStrategy = iota
	FillOne

	// FillForward and FillBackward carry the neighbouring value. Both are ordered
	// window functions, so both are pipeline breakers.
	FillForward
	FillBackward

	// FillMin, FillMax and FillMean fill with the column's own aggregate over the
	// WHOLE FRAME. That makes them pipeline breakers too, and worse ones: the
	// window sink holds every row, and it is one of the operators that is accounted
	// and refused under a memory limit rather than spilled.
	//
	// FillMean returns a FLOAT even for an integer column, because a mean is a
	// float. FillMin and FillMax preserve the type.
	FillMin
	FillMax
	FillMean
)

func (FillStrategy) String 

func (s FillStrategy) String() string

type GroupBy 

type GroupBy struct {
	// contains filtered or unexported fields
}

GroupBy is the intermediate handle between GroupBy and Agg.

func (*GroupBy) Agg 

func (g *GroupBy) Agg(exprs ...Expr) *LazyFrame

Agg reduces each group.

Every expression must be an aggregation: every column it reads has to pass through an aggregate function. `Col("a").Sum().Add(1)` is fine; `Col("a").Sum().Add(Col("b"))` is not, because b is still a per-row value and there is no defined way to combine the two shapes.

func (*GroupBy) Count 

func (g *GroupBy) Count() *LazyFrame

Count is shorthand for Agg(Len().Alias("count")).

func (*GroupBy) First 

func (g *GroupBy) First() *LazyFrame

func (*GroupBy) Last 

func (g *GroupBy) Last() *LazyFrame

func (*GroupBy) Len 

func (g *GroupBy) Len(name string) *LazyFrame

Len counts the rows in each group under the given name.

Distinct from Count, which uses the fixed name "count": Len takes one because a frame that already has a column called "count" is not unusual, and silently colliding would be worse than asking.

func (*GroupBy) MaintainOrder 

func (g *GroupBy) MaintainOrder() *GroupBy

MaintainOrder emits groups in first-appearance order.

Without it the group order is unspecified — which is standard (SQL uses set semantics and DuckDB exploits it for parallelism) but means a test that assumes an order is flaky.

The flag is no longer free, and that is the point of it

It USED to document a guarantee the implementation happened to give anyway. Since step 12 a group-by over more distinct keys than the memory limit holds partitions the overflow to disk, and the unordered path then emits partition-major: resident groups first, then partition 0's groups, then partition 1's. The order depends on where the freeze fell, so it depends on the limit.

The CONTENT never does. Every group, every value, identical at every limit. See WithMemoryLimit, which states the same contract from the other side.

Costs nothing when the aggregation fits in memory. When it spills it adds an int64 ordinal to every routed row and materialises the whole result to permute it — plan.Aggregate.MaintainOrder has the detail.

func (*GroupBy) Max 

func (g *GroupBy) Max() *LazyFrame

func (*GroupBy) Mean 

func (g *GroupBy) Mean() *LazyFrame

func (*GroupBy) Median 

func (g *GroupBy) Median() *LazyFrame

func (*GroupBy) Min 

func (g *GroupBy) Min() *LazyFrame

Min, Max, First, Last and NUnique apply to EVERY non-key column, because they select or count rather than compute.

func (*GroupBy) NUnique 

func (g *GroupBy) NUnique() *LazyFrame

func (*GroupBy) Quantile 

func (g *GroupBy) Quantile(q float64, interp Interpolation) *LazyFrame

Quantile applies one quantile to every numeric non-key column.

func (*GroupBy) Sum 

func (g *GroupBy) Sum() *LazyFrame

Sum, Mean, Min, Max, Median, NUnique, First and Last apply that aggregate to every non-key column. Sum, Mean, Median and Quantile apply to the NUMERIC non-key columns.

type Int128Value 

type Int128Value = i128.Int128

Int128Value is a signed 128-bit integer, the Go representation of an Int128 column. Read one with df.Column[ursus.Int128Value]("total").

type Interpolation 

type Interpolation = expr.Interpolation

Interpolation selects how Quantile resolves a rank falling between two values.

Aliased from the expression IR rather than redeclared, so the kernel and the public API cannot drift — the same arrangement JoinKind uses.

type Interval 

type Interval = dtype.Interval

Interval is a calendar-aware span: months, days and nanoseconds, kept apart because each obeys a different rule. See dtype.Interval for why a time.Duration cannot express "1 month" or, in a daylight-saving zone, "1 day".

func Every 

func Every(s string) Interval

Every parses an interval: "1y", "3mo", "1w", "2d", "6h", "15m", "30s", "1ns". Components may be concatenated ("1h30m") and a leading "-" negates the whole thing.

A parse failure is carried in the value rather than returned, and surfaces when the query is built — the deferred-error rule the rest of the library follows. Note "m" is minutes and "mo" is months.

func FromDuration 

func FromDuration(d time.Duration) Interval

FromDuration builds an ABSOLUTE interval. Deliberately not the same as the calendar spelling: FromDuration(24*time.Hour) is twenty-four hours, Every("1d") is one day, and on the two days a year a zone changes its offset those are different instants.

type JoinKind 

type JoinKind = plan.JoinKind

JoinKind is the shape of a join: which rows survive.

type JoinOption 

type JoinOption func(*joinCfg)

JoinOption configures a join.

func JoinCoalesce 

func JoinCoalesce(b bool) JoinOption

JoinCoalesce forces key merging on or off.

Unset is not the same as either: by default the key appears once for inner, left, right, semi and anti joins, and twice for a full join — which can leave the key null on either side, so there is no single side to take it from.

The default only merges keys that are named the SAME on both sides. Merging JoinLeftOn(Col("cust_id")) with JoinRightOn(Col("id")) would silently delete a column the caller named explicitly, so it takes an explicit JoinCoalesce(true).

func JoinHow 

func JoinHow(k JoinKind) JoinOption

JoinHow sets the join kind. Default JoinInner.

func JoinLeftOn 

func JoinLeftOn(keys ...Expr) JoinOption

JoinLeftOn and JoinRightOn name the key columns separately, for frames where they are called different things. They are used together and must agree in count; keys are paired positionally.

func JoinNullsEqual 

func JoinNullsEqual(b bool) JoinOption

JoinNullsEqual makes null keys match each other. Default false, which is SQL's rule and Polars': a null key matches nothing, including another null.

Note this is deliberately the OPPOSITE of GroupBy, where null forms its own group. Grouping asks "are these the same value?" and joining asks "is this the same entity?", and a missing identifier is not evidence of sameness.

func JoinOn 

func JoinOn(keys ...Expr) JoinOption

JoinOn names key columns present under the same name on both sides.

func JoinRightOn 

func JoinRightOn(keys ...Expr) JoinOption

func JoinSuffix 

func JoinSuffix(s string) JoinOption

JoinSuffix sets what is appended to a right-side column whose name collides with a left-side one. Default "_right".

func JoinValidate 

func JoinValidate(v JoinValidation) JoinOption

JoinValidate asserts a key cardinality, failing the query when the data violates it. 

orders.Join(customers, JoinOn(Col("customer_id")),
    JoinValidate(ursus.ValidateManyToOne))

Worth reaching for: an unexpected duplicate on the right multiplies every matching left row, and the result looks entirely plausible.

type JoinValidation 

type JoinValidation = plan.JoinValidation

JoinValidation asserts a cardinality and errors when the data violates it.

The first term names the LEFT frame: ManyToOne on orders.Join(customers) asserts that the customers are unique.

type LazyFrame 

type LazyFrame struct {
	// contains filtered or unexported fields
}

LazyFrame is an unexecuted query.

Immutable

Every builder method returns a NEW LazyFrame sharing an immutable plan. So 

base := ursus.Scan(src)
a := base.Filter(ursus.Col("x").Gt(5))
b := base.Filter(ursus.Col("x").Lt(0))

gives two independent queries, and `base` is unchanged. That idiom is the most common thing anyone does with a lazy frame, and it only works because nothing mutates in place. It also makes a LazyFrame safe to share between goroutines.

Sticky errors

A builder cannot return an error without destroying chaining, so errors are carried and surfaced at Collect (or read early via Err). An unknown column in Select does not panic and does not fail silently — it fails at Collect, with the available columns and a spelling suggestion.

func Concat 

func Concat(frames []*LazyFrame, opts ...ConcatOption) *LazyFrame

Concat stacks frames vertically, in the order given. 

ursus.Concat([]*ursus.LazyFrame{jan, feb, mar})
ursus.Concat(fs, ursus.WithConcatMode(ursus.ConcatDiagonal))

Why a slice rather than a variadic

It was `Concat(frames ...any)` for one release, which read better and let the mode be passed inline. It also made `ursus.Concat(a, "oops")` compile and fail at run time — in a library that spends generic methods and the Operand constraint precisely so that `Col("x").Gt(struct{}{})` does not. Ergonomics is not worth the one property the type system was being paid to provide.

For the common case the method form keeps both: LazyFrame.Concat is variadic and type-safe, so `jan.Concat(feb, mar)` needs no slice literal.

Order is preserved

Every row of the first frame precedes every row of the second. That is not free — it is why the operator drains children serially rather than racing them — and it is what lets Concat compose with Head, Tail and anything else that cares which rows come first.

Reconciliation happens in the plan

A frame missing a column, or holding one at a narrower type, is wrapped in a projection that supplies the null and performs the cast, so Explain shows why a column is null and projection pushdown still narrows each input independently.

func Frame 

func Frame(cols ...*Column) *LazyFrame

Frame builds an in-memory LazyFrame from columns.

This is the entry point for data that is already in Go — the counterpart to ScanParquet and friends, which arrive in step 2.

func FrameOf 

func FrameOf(cols ...*Column) (*LazyFrame, error)

FrameOf is Frame for callers who want the error separately.

func FromPlan 

func FromPlan(n plan.Node) *LazyFrame

FromPlan builds a LazyFrame over an existing logical plan. It exists so that a future SQL frontend can lower into the same IR and hand back a frame, without the root package having to import it — which would be a cycle.

func Scan 

func Scan(src plan.Source) *LazyFrame

Scan starts a query against a source.

func ScanCSV 

func ScanCSV(path string, opts ...CSVOption) *LazyFrame

ScanCSV reads a CSV file lazily.

Nothing is read until Collect, except the sample inference needs — and not even that if WithSchema is given. Projection reaches the reader, so selecting two columns of a hundred parses two. 

df, err := ursus.ScanCSV("sales.csv").
    Filter(ursus.Col("qty").Gt(10)).
    Select(ursus.Col("region"), ursus.Col("qty")).
    Collect(ctx)

func ScanCSVFiles 

func ScanCSVFiles(paths []string, opts ...CSVOption) *LazyFrame

ScanCSVFiles reads several files as one frame, in the order given.

func ScanCSVGlob 

func ScanCSVGlob(pattern string, opts ...CSVOption) *LazyFrame

ScanCSVGlob reads every file matching a shell pattern as one frame.

The files must share a schema; it is taken from the first match. Matches are sorted, so `part-*.csv` reads in the order the names imply rather than whatever the filesystem returns — row order is a property of the data here, and leaving it to readdir would make the same query return different orders on different machines.

func ScanCSVReader 

func ScanCSVReader(b []byte, name string, opts ...CSVOption) *LazyFrame

ScanCSVReader reads CSV from memory.

It takes the bytes rather than an io.Reader because a source is opened more than once — once for inference, once per execution — and a Reader cannot be rewound.

func ScanParquet 

func ScanParquet(path string, opts ...ParquetOption) *LazyFrame

ScanParquet reads a Parquet file lazily.

Only the footer is read until Collect. Projection reaches the reader, so a query selecting two columns of a hundred never decompresses the other ninety-eight, and a predicate on a column with statistics skips whole row groups. 

df, err := ursus.ScanParquet("events.parquet").
    Filter(ursus.Col("ts").Gt(cutoff)).
    Select(ursus.Col("user"), ursus.Col("ts")).
    Collect(ctx)

Flat schemas only. A file with a nested, repeated or encrypted column is refused with an error naming the column, rather than read with that column dropped.

func ScanParquetBytes 

func ScanParquetBytes(b []byte, name string, opts ...ParquetOption) *LazyFrame

ScanParquetBytes reads a Parquet file from memory.

func ScanParquetFiles 

func ScanParquetFiles(paths []string, opts ...ParquetOption) *LazyFrame

ScanParquetFiles reads several files as one frame, in the order given. The schema comes from the first.

func ScanParquetGlob 

func ScanParquetGlob(pattern string, opts ...ParquetOption) *LazyFrame

ScanParquetGlob reads every file matching a shell pattern as one frame, sorted by name so `part-*.parquet` reads in the order the names imply.

func (*LazyFrame) BottomK 

func (lf *LazyFrame) BottomK(k int, by ...SortKey) *LazyFrame

BottomK keeps the k smallest rows by the given keys. See TopK — including that nulls are placed last and so never appear in the result.

func (*LazyFrame) Collect 

func (lf *LazyFrame) Collect(ctx context.Context, opts ...CollectOption) (*DataFrame, error)

Collect runs the query and returns the whole result.

func (*LazyFrame) CollectBatches 

func (lf *LazyFrame) CollectBatches(ctx context.Context, opts ...CollectOption) iter.Seq2[*DataFrame, error]

CollectBatches streams the result without materialising the whole frame.

Breaking out of the loop tears the operator tree down, so an early exit is safe.

func (*LazyFrame) CollectInto 

func (lf *LazyFrame) CollectInto[T any](ctx context.Context, opts ...CollectOption) ([]T, error)

CollectInto runs the query and decodes each row into a T.

A generic METHOD, which is the Go 1.27 feature this library was waiting for: before, this had to be a package-level function taking the frame as an argument.

func (*LazyFrame) CollectSchema 

func (lf *LazyFrame) CollectSchema(ctx context.Context) (*Schema, error)

CollectSchema returns the result schema without reading any data.

It takes a context because resolving a source's schema is real I/O — a Parquet footer, a CSV sample. A schema request that could not be cancelled would be a hole in the cancellation story, which is why this is not the no-argument Schema() the design documents originally specified.

func (*LazyFrame) Concat 

func (lf *LazyFrame) Concat(others ...*LazyFrame) *LazyFrame

Concat stacks other frames beneath this one.

func (*LazyFrame) Count 

func (lf *LazyFrame) Count(ctx context.Context, opts ...CollectOption) (int64, error)

Count runs the query and returns the row count, retaining no data.

func (*LazyFrame) Drop 

func (lf *LazyFrame) Drop(names ...string) *LazyFrame

Drop removes columns by name.

Sugar over `Select(Exclude(...))`, which is the whole implementation: Exclude is already an expansion-time selector, so Drop inherits its error messages, its interaction with projection pushdown, and its behaviour on a name that is not there — the selector ignores it rather than failing, matching Polars.

func (*LazyFrame) DropNulls 

func (lf *LazyFrame) DropNulls(subset ...string) *LazyFrame

DropNulls removes rows that are null in any of the named columns, or in any column at all when no names are given.

It is `Filter(Col(c).IsNotNull(), ...)`, and being a filter rather than a dedicated node is what earns it Parquet row-group pruning: the pruner claims IsNotNull over a bare column, so a row group whose statistics say a column is entirely null is skipped without being read.

Note the asymmetry with a hypothetical DropNans: IsNotNull is TOTAL, so the predicate is never itself null and the filter's own null rule never comes into play. `Col(c).IsNotNan()` is null on a null row, and Filter drops null predicates, so the obvious spelling of DropNans would silently drop nulls too. That is why it is not here.

func (*LazyFrame) Err 

func (lf *LazyFrame) Err() error

Err returns the first deferred error, or nil.

func (*LazyFrame) Explain 

func (lf *LazyFrame) Explain(ctx context.Context, opts ...ExplainOption) (string, error)

Explain renders the query plan.

The output is stable and diff-friendly on purpose: it is what golden tests snapshot, and a plan diff is the only thing that notices a query which still returns the right answer while reading forty columns instead of two.

func (*LazyFrame) FillNan 

func (lf *LazyFrame) FillNan[T Operand](v T, subset ...string) *LazyFrame

FillNan replaces NaN in the named columns, or in every FLOAT column when no subset is given. Nulls are left alone — see Expr.FillNan.

func (*LazyFrame) FillNull 

func (lf *LazyFrame) FillNull[T Operand](v T, subset ...string) *LazyFrame

FillNull replaces nulls in the named columns, or in every column the value can fill when no subset is given. 

lf.FillNull(0)                  // every numeric column
lf.FillNull(0, "qty", "amount") // exactly these

Why the no-subset form restricts rather than refuses

Almost every real frame has a string column, and `lf.FillNull(0)` erroring on it would make the shorthand useless exactly where it is most wanted. So the value's own type chooses the columns — the same argument GroupBy(k).Sum()'s doc makes for restricting to numeric columns, and with the same consequence: it is not a silent skip, because the selector is part of the expression and Explain shows precisely which columns were filled.

A named column whose type the value cannot fill is a plan-time error, because the user asked for it by name.

The value is a WEAK literal, so filling an Int32 column with 0 leaves it Int32.

func (*LazyFrame) Filter 

func (lf *LazyFrame) Filter(preds ...Expr) *LazyFrame

Filter keeps rows where every predicate is true.

Predicates are AND-ed. A row whose predicate is NULL is DROPPED, matching SQL's WHERE. One consequence worth knowing: Filter(p) and Filter(p.Not()) do not partition the input, because a row where p is null is dropped by both.

func (*LazyFrame) GroupBy 

func (lf *LazyFrame) GroupBy(keys ...Expr) *GroupBy

GroupBy begins an aggregation. 

lf.GroupBy(ursus.Col("region")).
    Agg(ursus.Col("revenue").Sum(), ursus.Len().Alias("n"))

With no keys it is a GLOBAL aggregate producing exactly one row — including over an empty input, where `count(*)` is 0 rather than no rows at all.

func (*LazyFrame) GroupByDynamic 

func (lf *LazyFrame) GroupByDynamic(index Expr, o DynamicOptions) *GroupBy

GroupByDynamic groups rows into fixed temporal windows cut from a sorted index. 

lf.GroupByDynamic(ursus.Col("ts"), ursus.DynamicOptions{
        Every:   ursus.Every("1h"),
        GroupBy: []ursus.Expr{ursus.Col("service")},
    }).
    Agg(ursus.Len().Alias("n"))

It is not GroupBy(truncate(ts, every)), and the difference is empty windows

A hash group-by creates a group when a row arrives, so an hour with no rows simply is not in the output and a gap in a time series is invisible. A dynamic group generates the whole grid between the first and last instant, so the gap is a row with a zero in it. That is the entire reason this operator exists.

The index must be sorted, and it is CHECKED

Windows are cut as contiguous ranges of a sorted index, so an unsorted one gives wrong groups. ursus verifies rather than trusting an assertion: an unchecked SetSorted-style hint is a user claim that deletes a correctness check, and its failure mode is wrong rows with no error. Sort first if the check refuses.

func (*LazyFrame) HStack 

func (lf *LazyFrame) HStack(others ...*LazyFrame) *LazyFrame

HStack places frames side by side: the same rows, with the columns concatenated.

It buffers, and Concat does not

Nothing in the engine aligns batch boundaries across independent pipelines — one frame may deliver 8192 rows at a time while another delivers 100 — and pairing row i of each means having row i of each in hand. So every input is materialised. Concat, which appends rows rather than pairing them, streams.

Every frame must have the same height, and column names must not collide. There is no automatic suffixing: a join suffixes because it has a principled left and right to name the suffix after, and these inputs are peers.

func (*LazyFrame) Head 

func (lf *LazyFrame) Head(n int) *LazyFrame

Head keeps at most n rows.

func (*LazyFrame) Join 

func (lf *LazyFrame) Join(other *LazyFrame, opts ...JoinOption) *LazyFrame

Join combines this frame with another. 

enriched := orders.Join(customers,
    ursus.JoinOn(ursus.Col("customer_id")),
    ursus.JoinHow(ursus.JoinLeft),
)

Column names that appear on both sides get the right one suffixed; the join key appears once unless the kind is a full join. See JoinCoalesce and JoinSuffix.

func (*LazyFrame) JoinAsOf 

func (lf *LazyFrame) JoinAsOf(other *LazyFrame, opts ...AsOfOption) *LazyFrame

JoinAsOf matches each left row to the NEAREST right row rather than an equal one. 

trades.JoinAsOf(quotes,
    ursus.AsOfOn(ursus.Col("ts")),
    ursus.AsOfBy(ursus.Col("symbol")),
    ursus.AsOfTolerance(ursus.Every("1m")),
)

It is a LEFT join

Every left row survives; the right columns are null when nothing is near enough. So the output height always equals the left height, tolerance and strategy only decide which rows come back populated.

Both sides must be sorted on the as-of key, and it is CHECKED

The search is a binary search over a sorted run. ursus verifies rather than trusting an assertion — an unchecked sortedness hint is a user claim that deletes a correctness check, and its failure mode is wrong matches with no error. Sort first if the check refuses.

func (*LazyFrame) Limit 

func (lf *LazyFrame) Limit(n int) *LazyFrame

Limit is an alias for Head.

func (*LazyFrame) MergeSorted 

func (lf *LazyFrame) MergeSorted(other *LazyFrame, key string) *LazyFrame

MergeSorted interleaves this frame with another that is sorted on the same key, keeping the result sorted.

Not a concat: a concat appends, so the result is ordered only if the second frame begins after the first ends. Not a join either — nothing is matched and no row is dropped, so the output height is always the sum of the two.

Both schemas must match EXACTLY and both inputs must be sorted on key, which is checked rather than assumed.

func (*LazyFrame) Pipe 

func (lf *LazyFrame) Pipe(fn func(*LazyFrame) *LazyFrame) *LazyFrame

Pipe applies fn, for composing reusable query fragments.

func (*LazyFrame) Plan 

func (lf *LazyFrame) Plan() plan.Node

Plan returns the underlying logical plan.

func (*LazyFrame) Remove 

func (lf *LazyFrame) Remove(preds ...Expr) *LazyFrame

Remove is the inverse of Filter: it drops rows where ANY predicate is true.

Spelled out because the alternative reading is equally plausible: this is NOT(p1 OR p2 OR ...), i.e. a row survives only if every predicate is false. Rows where a predicate is null are dropped, same as Filter.

func (*LazyFrame) Rename 

func (lf *LazyFrame) Rename(names map[string]string) *LazyFrame

Rename changes column names, leaving everything else in place.

Sugar over `Select(All().MapName(...))`. MapName is the expansion-safe renamer — Alias sets ONE fixed name and so cannot apply to a multi-column selection, which is exactly why the two are different methods.

Names not present in the frame are ignored rather than an error, so a rename map written against a wider schema still works. Renaming two columns onto the same name IS an error, caught by Select's output-name uniqueness check with both offending expressions named.

func (*LazyFrame) Reverse 

func (lf *LazyFrame) Reverse() *LazyFrame

Reverse emits rows in the opposite order.

A full pipeline breaker — the last input row is the first output row — so it holds the whole frame. It does no comparison, which is what makes it cheaper than sorting by a synthesised descending index.

func (*LazyFrame) Rolling 

func (lf *LazyFrame) Rolling(index Expr, o RollingOptions) *GroupBy

Rolling gives every row its own window, reaching back Period from that row's own instant. The output has one row per input row.

Where GroupByDynamic answers "how many per hour", Rolling answers "how many in the hour before each event". The index must be sorted, and is checked.

func (*LazyFrame) Select 

func (lf *LazyFrame) Select(exprs ...Expr) *LazyFrame

Select computes a new set of columns, replacing the frame's columns entirely.

Expressions may expand: Select(All()) keeps everything, and Select(ColDType(Float64).Suffix("_f")) selects and renames every float column.

func (*LazyFrame) SinkCSV 

func (lf *LazyFrame) SinkCSV(ctx context.Context, path string, opts ...CSVSinkOption) error

SinkCSV runs the query and writes the result to path, streaming.

This is the half that makes "larger than RAM" true end to end. Collect holds the whole result; this holds one batch, so a query that reads more data than fits in memory and writes more data than fits in memory works.

The file is written to a temporary name and renamed on success, so a failed query leaves no half-written file where a complete one is expected.

func (*LazyFrame) SinkParquet 

func (lf *LazyFrame) SinkParquet(ctx context.Context, path string, opts ...ParquetSinkOption) error

SinkParquet runs the query and writes the result to path, streaming.

Memory is one row group rather than the whole result, so this is the write half of "larger than RAM". The file is written to a temporary name and renamed on success: a Parquet file without its footer is not merely truncated, it is unreadable, so leaving a partial one where a complete one is expected would be worse than leaving nothing.

func (*LazyFrame) Slice 

func (lf *LazyFrame) Slice(offset, length int) *LazyFrame

Slice keeps length rows starting at offset. A negative length means "to the end", which is how you drop a prefix without knowing the height.

Streaming: it counts rows past and stops, so it never holds more than one batch.

func (*LazyFrame) Sort 

func (lf *LazyFrame) Sort(keys ...SortKey) *LazyFrame

func (*LazyFrame) Tail 

func (lf *LazyFrame) Tail(n int) *LazyFrame

Tail keeps the last n rows.

Unlike Head it cannot stream to completion: which rows are the last n is unknown until the input ends, so it holds a ring of the most recent n. Bounded — O(n), not O(input) — but not free.

func (*LazyFrame) TopK 

func (lf *LazyFrame) TopK(k int, by ...SortKey) *LazyFrame

TopK keeps the k LARGEST rows by the given keys, and BottomK the k smallest.

Both are sugar, and that is the point

Each is `Sort(...).Head(k)`, which the limit-pushdown rule turns into a bounded top-k: kernel.ArgTopK is O(n log k) time and O(k) memory against a full sort's O(n log n) and O(n), and its contract is that it returns EXACTLY the indices ArgSort would, ties included. So there is no separate operator to keep in step with Sort, and `TopK(k)` and `Sort(...).Head(k)` cannot drift apart because they are the same plan.

Both place nulls LAST, unlike Sort

Sort keeps direction and null placement orthogonal on purpose — "`.Desc()` silently relocating the nulls surprises people every time" — and that argument does NOT transfer here. In a sort, placement is cosmetic: every row comes back either way. In a top-k it is SELECTION, and a null has no rank, so leaving the default (nulls first) would make `TopK(2)` over [3,1,4,1,5,null] return the null and the 5 — a row with no value in the very column being ranked.

So both force nulls last, and both say so. `Sort(...).Head(k)` remains available for a different placement, and it is one call away.

func (*LazyFrame) Unique 

func (lf *LazyFrame) Unique(subset ...string) *LazyFrame

Unique removes duplicate rows.

With no arguments it compares whole rows. With column names it compares only those, keeping the FIRST row for each distinct combination — so the other columns come from that row rather than being chosen arbitrarily.

func (*LazyFrame) VStack 

func (lf *LazyFrame) VStack(other *LazyFrame) *LazyFrame

VStack is Concat for exactly two frames, named for the operation people look for.

It is not cheaper than Concat. Polars documents vstack as "cheap (adds a chunk)", which relies on a chunked column layout ursus does not have — a Column here is one contiguous run, so stacking copies. Naming it the same and pretending otherwise would be the misleading part.

func (*LazyFrame) WithColumns 

func (lf *LazyFrame) WithColumns(exprs ...Expr) *LazyFrame

WithColumns adds columns, replacing any that already exist BY NAME and keeping their original position.

Later expressions can reference columns that earlier ones added, so `WithColumns(a.Alias("x"), Col("x").Mul(2))` works.

func (*LazyFrame) WithRowIndex 

func (lf *LazyFrame) WithRowIndex(name string, offset uint32) *LazyFrame

WithRowIndex prepends a Uint32 column numbering the rows from offset.

Row order is a property rather than an addressable label space, which is why there is no implicit index — this is how you ask for one when you want it.

It is a counter that crosses batch boundaries, so it depends on batches arriving in input order, and it is deliberately not the same thing as `CumCount(false).Over()`: that computes identical numbers through the window sink, which buffers the entire frame to do it.

func (*LazyFrame) WriteCSV 

func (lf *LazyFrame) WriteCSV(ctx context.Context, w io.Writer, opts ...CSVSinkOption) error

WriteCSV runs the query and writes the result to w, streaming.

It does not close w. Whoever opened it closes it — the only rule that works when the destination might be os.Stdout.

func (*LazyFrame) WriteParquet 

func (lf *LazyFrame) WriteParquet(ctx context.Context, w io.Writer, opts ...ParquetSinkOption) error

WriteParquet runs the query and writes the result to w, streaming. It does not close w.

type Literal 

type Literal interface {
	~bool |
		~int | ~int8 | ~int16 | ~int32 | ~int64 |
		~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 |
		~float32 | ~float64 |
		~string | ~[]byte |
		time.Time
}

Literal is the set of Go types that lift to a column literal.

Every accepted width is enumerated explicitly because Go infers a type parameter from the argument's type with NO implicit widening: `a.Gt(int32(5))` is a compile error unless ~int32 appears here.

time.Duration is DELIBERATELY ABSENT. Its underlying type is int64, so listing it alongside ~int64 gives overlapping type sets and the union does not compile — this is the defect the design review found. ~int64 already admits it, and a type switch recovers the named type exactly, so nothing is lost.

time.Time is a bare (non-~) term: it is a defined struct type, disjoint from every other term. A union term may be a type with methods; only interface terms with methods are forbidden.

type MappingStrategy 

type MappingStrategy = expr.MappingStrategy

MappingStrategy decides how a window's per-partition result returns to the frame.

type MemoryStats 

type MemoryStats struct {
	// Peak is the largest total the query retained at one moment, in bytes.
	//
	// It counts what buffering operators HOLD ACROSS BATCHES, deduplicated by
	// allocation so a payload two operators share is counted once. It does not
	// count a kernel's transient output, because ursus's allocator reports
	// nothing and there is no hook at the allocation site.
	Peak int64

	// Limit is the ceiling that was in force, or 0 if there was none.
	Limit int64

	// Spills is how many files a spilling operator wrote: sorted runs for a sort,
	// radix partitions for a group-by, and for a join BOTH sides' partitions —
	// counted at every level of the recursion.
	//
	// It is a count of FILES, not of bytes and not of distinct spill events, and a
	// group-by that recursed reports more than one level's fan-out — which is the
	// only public evidence that a partition did not fit on the first try.
	Spills int64
}

MemoryStats reports what a query actually held.

Peak is the number that demonstrates bounded memory: that a query FINISHED proves only that it did not run out, while a peak far below the input size proves it never held the input at all.

type Operand 

type Operand interface {
	Expr | Literal
}

Operand is either another Expr or a Go scalar that lifts to a literal.

This union — a struct type alongside an embedded constraint interface — is what lets every binary operator accept both forms with no wrapper at the call site.

type ParquetOption 

type ParquetOption func(*parquet.Options)

ParquetOption configures ScanParquet.

func WithPruning 

func WithPruning(b bool) ParquetOption

WithPruning enables or disables row-group skipping from column statistics.

It is on by default. The switch exists because pruning is the one part of the Parquet reader that can change which rows come back, so a wrong answer must be bisectable to it — the same reason plan.Flags can disable each optimizer rule.

type ParquetSinkOption 

type ParquetSinkOption interface {
	// contains filtered or unexported methods
}

ParquetSinkOption is anything SinkParquet and WriteParquet accept: a writer option (WithCompression, WithRowGroupRows, WithStatistics) or an execution option (WithMemoryLimit, WithSpillDir, WithBatchSize, WithThreads).

A sink is where a memory limit matters most — it is the consumer that streams, so it is the one a larger-than-RAM query ends in — and before this the two option families could not meet.

It is an interface rather than a variadic `...any` because the union has to be checked at compile time. A library that uses generic methods and an Operand constraint to stop `Col("x").Gt(struct{}{})` from compiling should not then accept SinkParquet(ctx, path, "oops").

type ParquetWriteOption 

type ParquetWriteOption func(*parquet.WriteOptions)

ParquetWriteOption configures the Parquet writer.

func WithCompression 

func WithCompression(c compress.Compression) ParquetWriteOption

WithCompression sets the codec applied to every column. Default Snappy.

func WithRowGroupRows 

func WithRowGroupRows(n int) ParquetWriteOption

WithRowGroupRows sets the target row-group size. It is the unit of both pruning granularity and writer memory: smaller groups prune better and buffer less.

func WithStatistics 

func WithStatistics(b bool) ParquetWriteOption

WithStatistics enables or disables column statistics. On by default — a file without them cannot be pruned, which gives up the main reason to use Parquet.

type RankMethod 

type RankMethod = expr.RankMethod

RankMethod decides how Rank breaks ties.

type RollingOptions 

type RollingOptions struct {
	// Period is how far back each row's window reaches from its own instant.
	Period Interval

	// Offset shifts the window's end away from the row's own instant.
	Offset Interval

	// Closed says which end belongs to the window. The zero value is ClosedRight —
	// (end-period, end] — because a rolling window ends AT the row, so closing the
	// left end instead would drop every row from its own window.
	Closed Closed

	// GroupBy adds categorical keys, as in DynamicOptions.
	GroupBy []Expr
}

RollingOptions configures Rolling.

type Schema 

type Schema = dtype.Schema

Schema is an ordered, name-unique sequence of Fields.

func MustSchema 

func MustSchema(fields ...Field) *Schema

MustSchema is NewSchema for tests and package-level vars.

func NewSchema 

func NewSchema(fields ...Field) (*Schema, error)

NewSchema builds a schema, rejecting duplicate column names.

type Series 

type Series[T any] = data.Series[T]

Series is a typed view over a Column.

func TypedColumn 

func TypedColumn[T any](c *Column) (*Series[T], error)

TypedColumn returns a typed view over a column.

type SortKey 

type SortKey struct {
	// contains filtered or unexported fields
}

SortKey is one component of an ordering. Build one with Asc or Desc.

func Asc 

func Asc(e Expr) SortKey

Asc orders ascending. Nulls come FIRST by default; call NullsLast to move them.

func Desc 

func Desc(e Expr) SortKey

Desc orders descending, with nulls still FIRST by default.

Null placement is deliberately independent of direction. SQL ties them together — nulls sort as the largest value, so they land last ascending and first descending — which means switching to Desc silently relocates them. Keeping the two orthogonal is more predictable.

func (SortKey) NullsFirst 

func (k SortKey) NullsFirst() SortKey

NullsFirst places nulls at the start of the output.

func (SortKey) NullsLast 

func (k SortKey) NullsLast() SortKey

NullsLast places nulls at the END of the output, whichever direction the key sorts in.

type Span 

type Span interface {
	Interval | time.Duration
}

Span is an interval in either spelling — the calendar-aware Interval, or a plain time.Duration for the absolute case.

The same union shape as Operand, and for the same reason: it lets one method accept both forms with no wrapper at the call site, so `Truncate(time.Hour)` keeps compiling now that `Truncate(Every("1mo"))` is also legal.

type StrExpr 

type StrExpr struct {
	// contains filtered or unexported fields
}

StrExpr is the string namespace: `Col("name").Str().ToLower()`.

A small wrapper struct whose methods return Expr, so chaining continues naturally through it. This is the shape ursus-api.md §4.5 specifies for every namespace, and `.dt` follows it identically.

The literal flag

Contains, Find, CountMatches, Replace and ReplaceAll take `literal bool`, and literal is what you want unless you know otherwise. Go's regexp is RE2: no backtracking, so no catastrophic blowup, but also no JIT — a plain substring test through it costs far more than strings.Contains. Passing literal=true keeps the common case on the fast path, which is why the flag is explicit rather than inferred from whether the pattern looks like a regex.

func (StrExpr) Contains 

func (s StrExpr) Contains(pattern string, literal bool) Expr

Contains reports whether each value contains pattern.

func (StrExpr) CountMatches 

func (s StrExpr) CountMatches(pattern string, literal bool) Expr

CountMatches counts non-overlapping occurrences.

func (StrExpr) EndsWith 

func (s StrExpr) EndsWith(suffix string) Expr

func (StrExpr) Extract 

func (s StrExpr) Extract(pattern string, group int) Expr

Extract returns capture group n of the first match, or NULL if there is none. Group 0 is the whole match. Regex only: extracting a literal is just Find.

func (StrExpr) Find 

func (s StrExpr) Find(pattern string, literal bool) Expr

Find returns the byte offset of the first match, or NULL when there is none.

Null rather than -1: a sentinel index would compare and do arithmetic like a real position, so `find(x) < 5` would be true for "not found".

func (StrExpr) Head 

func (s StrExpr) Head(n int) Expr

Head and Tail are Slice's common cases.

func (StrExpr) LenBytes 

func (s StrExpr) LenBytes() Expr

LenBytes counts bytes; LenChars counts runes. They differ on any non-ASCII input, and which one is wanted is not guessable — so there is no `Len`.

func (StrExpr) LenChars 

func (s StrExpr) LenChars() Expr

func (StrExpr) Replace 

func (s StrExpr) Replace(pattern, value string, literal bool) Expr

Replace substitutes the FIRST match; ReplaceAll substitutes every match.

func (StrExpr) ReplaceAll 

func (s StrExpr) ReplaceAll(pattern, value string, literal bool) Expr

func (StrExpr) Reverse 

func (s StrExpr) Reverse() Expr

Reverse reverses by RUNE, not by byte.

func (StrExpr) Slice 

func (s StrExpr) Slice(offset, length int) Expr

Slice takes length runes from offset, which may be negative to count from the end. Runes rather than bytes, so it can never split a character in half.

func (StrExpr) StartsWith 

func (s StrExpr) StartsWith(prefix string) Expr

StartsWith and EndsWith are always literal — a prefix match against a regex is not a well-defined thing to ask for.

func (StrExpr) StripChars 

func (s StrExpr) StripChars(chars string) Expr

StripChars trims any of the given characters from both ends. An empty set trims whitespace, matching Polars.

func (StrExpr) StripPrefix 

func (s StrExpr) StripPrefix(prefix string) Expr

func (StrExpr) StripSuffix 

func (s StrExpr) StripSuffix(suffix string) Expr

func (StrExpr) Tail 

func (s StrExpr) Tail(n int) Expr

func (StrExpr) ToDate 

func (s StrExpr) ToDate() Expr

func (StrExpr) ToDatetime 

func (s StrExpr) ToDatetime(unit dtype.TimeUnit, tz string) Expr

func (StrExpr) ToInteger 

func (s StrExpr) ToInteger() Expr

ToInteger, ToDate and ToDatetime parse. They are ordinary casts, which is why they are spelled as casts rather than as new kernels — and unparseable values become NULL rather than failing the query.

func (StrExpr) ToLower 

func (s StrExpr) ToLower() Expr

func (StrExpr) ToUpper 

func (s StrExpr) ToUpper() Expr

type ThenBuilder 

type ThenBuilder struct {
	// contains filtered or unexported fields
}

ThenBuilder is a conditional that may take another When or be closed by Otherwise.

func (ThenBuilder) Otherwise 

func (t ThenBuilder) Otherwise[T Operand](v T) Expr

Otherwise supplies the fall-through value and closes the chain.

The chain becomes a right-nested tree of conditionals, in the order written: when(c1).then(v1).when(c2).then(v2).otherwise(d) is Cond{c1, v1, Cond{c2, v2, d}}. Nesting rather than a flat list is what keeps type unification well-defined — promotion is pairwise, and folding a flat list would make the result depend on an association order the user never chose.

func (ThenBuilder) When 

func (t ThenBuilder) When(pred Expr) WhenBuilder

When adds another condition, the else-if of the chain.

type TimeUnit 

type TimeUnit = dtype.TimeUnit

TimeUnit is the resolution of a Time, Datetime or Duration.

type WhenBuilder 

type WhenBuilder struct {
	// contains filtered or unexported fields
}

WhenBuilder is a conditional awaiting its Then.

func When 

func When(pred Expr) WhenBuilder

When begins a conditional. 

ursus.When(ursus.Col("score").Ge(90)).Then("A").
    When(ursus.Col("score").Ge(80)).Then("B").
    Otherwise("F").Alias("grade")

Why a builder and not a function

The alternative was a variadic Case(cond1, val1, cond2, val2, …, default). That reads compactly and pairs its arguments POSITIONALLY, so a miscounted list is a runtime error rather than a type error — and the compiler cannot tell a condition from a value when both are Expr. The builder makes each pair a method call, so there is nothing to miscount.

An unterminated chain cannot be used, by construction

Only Otherwise returns an Expr; WhenBuilder and ThenBuilder are not expressions and satisfy nothing that takes one. So a chain that forgets its else branch fails to compile rather than silently defaulting. Where a null default IS wanted, say so: Otherwise(ursus.Null(ursus.NullT)).

Both branches are evaluated

This is a columnar engine, so Then and Otherwise are each computed for every row and then merged. That is unobservable — expressions have no side effects, and an arithmetic fault such as division by zero yields NULL rather than trapping — but it does mean a conditional does not make an expensive branch cheaper.

func (WhenBuilder) Then 

func (w WhenBuilder) Then[T Operand](v T) ThenBuilder

Then supplies the value for the pending condition. The value lifts, so Then("A") needs no Lit.

type WindowSpec 

type WindowSpec struct {
	PartitionBy []Expr
	OrderBy     []SortKey
	Mapping     MappingStrategy
}

WindowSpec is the full form of a window, for OverWith.

Source Files

View all Source files

Directories

Path Synopsis
Package dtype is ursus's type system: DataType, Field and Schema.
 Package dtype is ursus's type system: DataType, Field and Schema.
Package i128 provides a signed 128-bit integer.
 Package i128 provides a signed 128-bit integer.
internal
arrowx
Package arrowx is the only package in ursus that imports arrow-go.
 Package arrowx is the only package in ursus that imports arrow-go.
bitmap
Package bitmap is ursus's validity-bitmap layer.
 Package bitmap is ursus's validity-bitmap layer.
data
Package data holds ursus's runtime column representation.
 Package data holds ursus's runtime column representation.
exec
Package exec drives a physical operator tree to completion.
 Package exec drives a physical operator tree to completion.
execopt
Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
 Package execopt carries the execution-time knobs that are not planning decisions: how much memory one query may hold, and where it is allowed to spill.
expr
Package expr is ursus's expression IR.
 Package expr is ursus's expression IR.
gen/levels command
Command levels enforces ursus's import-level invariant:
 Command levels enforces ursus's import-level invariant:
kernel
Package kernel is ursus's compute layer.
 Package kernel is ursus's compute layer.
physical
Package physical turns a logical plan into runnable operators, and expressions into columns.
 Package physical turns a logical plan into runnable operators, and expressions into columns.
plan
Package plan is ursus's logical plan IR.
 Package plan is ursus's logical plan IR.
source
Package source is the runtime half of ursus's scan contract.
 Package source is the runtime half of ursus's scan contract.
source/csv
Package csv reads and writes delimited text files.
 Package csv reads and writes delimited text files.
source/memsrc
Package memsrc is an in-memory scan source.
 Package memsrc is an in-memory scan source.
source/parquet
Package parquet reads and writes Apache Parquet files.
 Package parquet reads and writes Apache Parquet files.
source/testsrc
Package testsrc is a scan source that lies about its capabilities on purpose.
 Package testsrc is a scan source that lies about its capabilities on purpose.
spill
Package spill writes batches to a file and reads them back exactly.
 Package spill writes batches to a file and reads them back exactly.
uerr
Package uerr is ursus's error type.
 Package uerr is ursus's error type.
Package ursustest provides assertions for testing code that uses ursus.
 Package ursustest provides assertions for testing code that uses ursus.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.