compute

package
v0.1.8 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 25, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package compute holds the vectorized kernel library. Each kernel takes one or more Series and returns a new Series (or a scalar). Kernels preserve the input Series' lifecycle; callers are responsible for releasing inputs and the returned result.

Null semantics. Kernels propagate nulls per polars' defaults. For arithmetic and comparison, if any input at position i is null, the output at i is null. Logical kernels use Kleene three-valued logic. Aggregates skip nulls by default.

Parallelism. Kernels use internal/pool.ParallelFor over the row range. Row-range parallelism is disabled for small inputs (see minParallelRows) to avoid goroutine overhead.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrDTypeMismatch    = errors.New("compute: dtype mismatch")
	ErrLengthMismatch   = errors.New("compute: length mismatch")
	ErrUnsupportedDType = errors.New("compute: unsupported dtype for kernel")
	ErrDivisionByZero   = errors.New("compute: integer division by zero")
)

Sentinel errors returned by compute kernels.

View Source 
var ErrMaskNotBool = fmt.Errorf("%w: filter mask must be Bool", ErrDTypeMismatch)

ErrMaskNotBool indicates a filter mask was not a boolean Series.

View Source 
var MaxInt64PairFold func(buf, col []int64) int

MaxInt64PairFold is the exported hook used by dataframe.rowReduceInt64. When the CPU has AVX2 the SIMD kernel runs; scalar fallback covers pre-AVX2 hosts and any remainder past the SIMD-aligned prefix.

Set at init() time in rowreduce_register_amd64.go.

View Source 
var MaxInt64PairFoldNT func(buf, col []int64) int

MaxInt64PairFoldNT / MinInt64PairFoldNT are the non-temporal-store variants. Callers should dispatch to these when the output buffer is too large to stay in L2 (n × 8 bytes > ~L2 / 2).

View Source 
var MinInt64PairFold func(buf, col []int64) int

MinInt64PairFold is the exported min hook.

View Source 
var MinInt64PairFoldNT func(buf, col []int64) int

Functions

func Add 

func Add(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Add returns a Series with elementwise a + b. a and b must have the same length and dtype. Nulls propagate: if a[i] or b[i] is null, result[i] is null.

func AddLit 

func AddLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

AddLit / SubLit / MulLit / DivLit compute a ∘ lit elementwise without materialising lit as a full broadcast Series. Halves the memory traffic vs Add/Sub/Mul/Div for the `col OP scalar` case that dominates query patterns like `pl.col("x") * 2`.

func And 

func And(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

And returns a boolean Series a AND b with Kleene three-valued logic:

  • true AND null = null
  • false AND null = false
  • null AND null = null

func Cast 

func Cast(ctx context.Context, s *series.Series, to dtype.DType, opts ...Option) (*series.Series, error)

Cast returns a Series whose values are s converted to the target dtype.

Semantics follow polars' strict=false default:

  • Integer widening (e.g. i32 -> i64) never fails.
  • Integer narrowing (e.g. i64 -> i32) produces null when the value does not fit in the target range.
  • Float-to-integer truncates; values outside the target range produce null.
  • Integer-to-float is exact for 32-bit integers; f64 may lose precision for large i64.
  • Numeric-to-string uses strconv's default formatting.
  • String-to-numeric uses strconv.Parse*; unparseable values produce null.
  • Bool-to-integer maps false=0, true=1; bool-to-float likewise.

Casting to the same dtype returns a clone of the input.

func Count 

func Count(s *series.Series) int

Count returns the number of non-null values in s.

func Div 

func Div(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Div returns a Series with elementwise a / b. For integer dtypes, division by zero produces a null in the output. For floats, zero divisors follow IEEE 754 (inf, -inf, nan).

func DivLit 

func DivLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

func Eq 

func Eq(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Eq returns a boolean Series a == b. Nulls propagate.

func EqLit 

func EqLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

EqLit is the ==-with-literal variant.

func Filter 

func Filter(ctx context.Context, s, mask *series.Series, opts ...Option) (*series.Series, error)

Filter returns a Series containing only the positions i where mask[i] is true. Null mask entries are treated as false.

func FusedFilterFloat64ByBitmap 

func FusedFilterFloat64ByBitmap(name string, src []float64, maskBytes []byte, n int, mem memory.Allocator) (*series.Series, error)

FusedFilterFloat64ByBitmap mirrors FusedFilterInt64ByBitmap for float64.

func FusedFilterInt64ByBitmap 

func FusedFilterInt64ByBitmap(name string, src []int64, maskBytes []byte, n int, mem memory.Allocator) (*series.Series, error)

fusedFilterInt64 scans the mask bitmap and writes surviving src values directly into the output buffer in one pass. Processes 64 bits at a time via POPCNT (sizing) and TZCNT (gather). For n>=256K we parallelize: 1) per-chunk POPCNT to compute chunk output sizes, 2) exclusive prefix-sum → per-chunk output offset, 3) parallel per-chunk scatter into pre-sized output. Mirrors polars' bitmap-aware filter: one allocation, sequential reads, no intermediate []int indices. FusedFilterInt64ByBitmap is the exported entry point for kernels (e.g. DropNulls) that already own a packed bitmap matching the mask semantics expected by fusedFilterInt64. Skips the mask wrap and the Filter dispatch; writes straight to an arrow-backed buffer.

func Ge 

func Ge(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Ge returns a boolean Series a >= b. Nulls propagate.

func GeLit 

func GeLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

GeLit is the >=-with-literal variant.

func Gt 

func Gt(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Gt returns a boolean Series a > b. Nulls propagate.

func GtLit 

func GtLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

GtLit returns a boolean Series a > lit. Scalar literal comparison avoids broadcasting the literal into a full-length Series: 2× faster and half the memory. Polars' `col > 123` expression compiles to a similar kernel. Supported dtypes: int64, float64. Other dtypes fall back to the broadcast form internally for correctness.

func IsNotNull 

func IsNotNull(ctx context.Context, s *series.Series, opts ...Option) (*series.Series, error)

IsNotNull returns a boolean Series where out[i] is true iff s[i] is valid.

func IsNull 

func IsNull(ctx context.Context, s *series.Series, opts ...Option) (*series.Series, error)

IsNull returns a boolean Series where out[i] is true iff s[i] is null. The result itself has no nulls.

func Le 

func Le(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Le returns a boolean Series a <= b. Nulls propagate.

func LeLit 

func LeLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

LeLit is the <=-with-literal variant.

func Lt 

func Lt(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Lt returns a boolean Series a < b. Nulls propagate.

func LtLit 

func LtLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

LtLit is the <-with-literal variant. See GtLit.

func MaxFloat64 

func MaxFloat64(ctx context.Context, s *series.Series, opts ...Option) (float64, bool, error)

func MaxInt64 

func MaxInt64(ctx context.Context, s *series.Series, opts ...Option) (int64, bool, error)

MaxInt64 returns the maximum non-null integer value as int64.

func MeanFloat64 

func MeanFloat64(ctx context.Context, s *series.Series, opts ...Option) (float64, bool, error)

MeanFloat64 returns the arithmetic mean of non-null values. Returns (NaN, false, nil) when the Series is empty or fully null.

func MinFloat64 

func MinFloat64(ctx context.Context, s *series.Series, opts ...Option) (float64, bool, error)

MinFloat64, MaxFloat64 are the float analogues. NaN values participate in ordering in a deterministic way: NaN is treated as greater than all non-NaN values, matching polars' default ordering.

func MinInt64 

func MinInt64(ctx context.Context, s *series.Series, opts ...Option) (int64, bool, error)

MinInt64 returns the minimum non-null integer value as int64. The bool return is false when the Series is empty or fully null.

func Mul 

func Mul(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Mul returns a Series with elementwise a * b.

func MulLit 

func MulLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

func Ne 

func Ne(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Ne returns a boolean Series a != b. Nulls propagate.

func NeLit 

func NeLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

NeLit is the !=-with-literal variant.

func Not 

func Not(ctx context.Context, s *series.Series, opts ...Option) (*series.Series, error)

Not returns a boolean Series NOT a. Nulls propagate.

func NullCount 

func NullCount(s *series.Series) int

NullCount returns the number of null values in s.

func Or 

func Or(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Or returns a boolean Series a OR b with Kleene three-valued logic:

  • false OR null = null
  • true OR null = true
  • null OR null = null

func PoolingMem 

func PoolingMem(userMem memory.Allocator) memory.Allocator

PoolingMem is the exported form for dataframe / lazy / series callers.

func Sort 

func Sort(ctx context.Context, s *series.Series, so SortOptions, opts ...Option) (*series.Series, error)

Sort returns a new Series with values arranged by a stable sort.

Fast paths: when s has no nulls and a simple primitive dtype, Sort skips the index-sort machinery and sorts the raw []T with a typed pdqsort. Benchmarked at ~80x the throughput of the general path for int64.

func SortIndices 

func SortIndices(ctx context.Context, s *series.Series, so SortOptions, opts ...Option) ([]int, error)

SortIndices returns the stable permutation that would sort s. The result is a []int of length s.Len(). Apply it via Take to materialize the sorted Series or to permute other columns.

func SortIndicesMulti 

func SortIndicesMulti(ctx context.Context, cols []*series.Series, opts []SortOptions, kernelOpts ...Option) ([]int, error)

SortIndicesMulti returns the stable permutation that would sort by the given columns with per-column options. All columns must have the same length as cols[0].

func Sub 

func Sub(ctx context.Context, a, b *series.Series, opts ...Option) (*series.Series, error)

Sub returns a Series with elementwise a - b.

func SubLit 

func SubLit(ctx context.Context, a *series.Series, lit any, opts ...Option) (*series.Series, error)

func SumFloat64 

func SumFloat64(ctx context.Context, s *series.Series, opts ...Option) (float64, error)

SumFloat64 returns the sum of non-null values for float dtypes.

func SumInt64 

func SumInt64(ctx context.Context, s *series.Series, opts ...Option) (int64, error)

SumInt64 returns the sum of non-null values. Overflows wrap per Go int64 semantics, matching polars-core behavior for integer sums. Only supports integer dtypes; use SumFloat64 for floats.

func Take 

func Take(ctx context.Context, s *series.Series, indices []int, opts ...Option) (_ *series.Series, err error)

Take returns a Series assembled by gathering rows at the given indices. Negative or out-of-range indices return an error. A null at indices[i] position is not possible (the caller provides plain ints); to materialize a null use the mask form via Filter or rely on gather primitives in the expression layer.

func Where 

func Where(ctx context.Context, cond, ifTrue, ifFalse *series.Series, opts ...Option) (*series.Series, error)

Where implements polars' when(cond).then(ifTrue).otherwise(ifFalse). For each row:

  • if cond is null or false -> take ifFalse[row]
  • if cond is true -> take ifTrue[row]

A null mask entry is treated as false (same as polars). The three inputs must share length. Dtypes must match between ifTrue and ifFalse; the output dtype matches them.

Nulls in ifTrue/ifFalse propagate: if the picked side is null, the output position is null.

Types

type NullPosition 

type NullPosition uint8

NullPosition controls where null values sort relative to non-null values. 

const (
	// NullsLast places null values after all non-null values (polars default
	// for ascending sorts).
	NullsLast NullPosition = iota
	// NullsFirst places null values before all non-null values.
	NullsFirst
)

type Numeric 

type Numeric interface {
	~int8 | ~int16 | ~int32 | ~int64 |
		~uint8 | ~uint16 | ~uint32 | ~uint64 |
		~float32 | ~float64
}

Numeric is the set of dtypes that participate in arithmetic kernels.

type Option 

type Option func(*config)

Option configures a kernel call.

func WithAllocator 

func WithAllocator(alloc memory.Allocator) Option

WithAllocator overrides the memory allocator used by a kernel.

func WithName 

func WithName(name string) Option

WithName overrides the name of the output Series. By default, binary kernels inherit the name of the left operand.

func WithParallelism 

func WithParallelism(n int) Option

WithParallelism caps the number of goroutines used by a kernel. Zero or negative values use the process default (GOMAXPROCS).

type Ordered 

type Ordered interface {
	~int8 | ~int16 | ~int32 | ~int64 |
		~uint8 | ~uint16 | ~uint32 | ~uint64 |
		~float32 | ~float64 | ~string
}

Ordered is the set of dtypes that support ordering comparisons.

type SortOptions 

type SortOptions struct {
	Descending bool
	Nulls      NullPosition
}

SortOptions tune the sort behavior.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.