Help us shape the future of Go! Take the 2026 Go Developer Survey

enchanter

package module
v0.5.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 25, 2026 License: MIT Imports: 5 Imported by: 0

README

Enchanter 🧙‍♂️

Go Reference CI

Formerly known as aargh — renamed in v0.2.0. The v0.1.x releases remain available under the old module path.

Enchanter is a data wrangling library in pure Go, in the spirit of Pandas and Polars in Python and dplyr in R: typed, nullable columns (series) composed into DataFrames with select / filter / group by / join / sort / aggregate pipelines. Series and DataFrames interoperate with Apache Arrow, which also powers the Parquet and Arrow IPC readers and writers.

Enchanter is a work in progress and the API is not stable yet.

Install

go get github.com/caerbannogwhite/enchanter

Requires Go 1.26+. Pure Go, no cgo.

Quick start

package main

import (
	"strings"

	"github.com/caerbannogwhite/enchanter"
	"github.com/caerbannogwhite/enchanter/dataframe"
)

func main() {
	data1 := `
name,age,weight,junior,department,salary band
Alice C,29,75.0,F,HR,4
John Doe,30,80.5,true,IT,2
Bob,31,85.0,F,IT,4
Jane H,25,60.0,false,IT,4
Mary,28,70.0,false,IT,3
Oliver,32,90.0,true,HR,1
Ursula,27,65.0,f,Business,4
Charlie,33,60.0,t,Business,2
Megan,26,55.0,F,IT,3`

	dataframe.ReadCsv(enchanter.NewContext()).
		SetReader(strings.NewReader(data1)).
		Read().
		Select("department", "age", "weight", "junior").
		GroupBy("department").
		Agg(dataframe.Min("age"), dataframe.Max("weight"), dataframe.Mean("junior"), dataframe.Count()).
		Run().
		PPrint(dataframe.NewPPrintParams().SetUseLipGloss(true))
}

//   DataFrame: 3 rows, 5 columns
// ╭────────────┬──────────┬─────────────┬──────────────┬───────╮
// │ department │ min(age) │ max(weight) │ mean(junior) │ n     │
// ├────────────┼──────────┼─────────────┼──────────────┼───────┤
// │ String     │ Float64  │ Float64     │ Float64      │ Int64 │
// ├────────────┼──────────┼─────────────┼──────────────┼───────┤
// │ HR         │    29.00 │       90.00 │       0.5000 │ 2.000 │
// │ IT         │    25.00 │       85.00 │       0.2000 │ 5.000 │
// │ Business   │    27.00 │       65.00 │       0.5000 │ 2.000 │
// ╰────────────┴──────────┴─────────────┴──────────────┴───────╯

More runnable examples live in examples and in the package documentation on pkg.go.dev.

Reading and writing files

Format Read Write Notes
CSV ✅ ✅ delimiter, header, type guessing
Parquet ✅ ✅ via Apache Arrow; types + nulls preserved
Arrow IPC (Feather) ✅ ✅
XPT (SAS transport) ✅ ✅ versions 5–9
XLSX ✅ ✅
JSON ✅ ✅ record-oriented
HTML ✅ ✅ tables
Markdown ✅ ✅ tables
SAS7BDAT ✅ — read-only, via kshedden/datareader

All readers and writers share the same builder style:

// Parquet round trip: types and nulls survive, unlike CSV.
err := df.WriteParquet().SetPath("people.parquet").Write()

df2 := dataframe.ReadParquet(ctx).
	SetPath("people.parquet").
	Read()

See examples/parquet for a runnable version.

Apache Arrow interop

Every series can produce an Arrow array and every DataFrame an Arrow record batch, which makes Enchanter data exchangeable with anything that speaks Arrow (DuckDB, Polars, pandas, DataFusion, Spark, ...):

rec := df.ToArrowRecord() // freshly built, owned by the record
defer rec.Release()       // optional under the default GC-backed allocator

df2 := dataframe.NewDataFrameFromArrowRecord(rec, ctx)

Conversion notes:

  • Enchanter null masks map onto Arrow validity bitmaps in both directions.
  • series.ArrowArrayToSeries accepts more Arrow types than Enchanter has series types (all integer widths, Float32, Date32/64, every timestamp unit, LargeString, ...) and funnels them onto the native types: integers normalize to int64, timestamps to nanoseconds.
  • Conversions are materialized copies — no reference to the source Arrow memory is retained, so you may release inputs immediately.
  • Releasing values returned to you is optional under GC-backed allocators (the default); see Context.Allocator for the exact ownership contract.

Data types

Seven nullable, columnar series types. Nulls live in a separate bit-packed mask, so a column with no nulls carries no null overhead.

Series Go element type
Bools bool
Ints int
Int64s int64
Float64s float64
Strings *string (interned via a shared pool)
Times time.Time
Durations time.Duration

Not implemented (would be added on demand): narrower integers (Int8/16/32), Float32, complex numbers, and a bit-packed memory-optimized Bool.

Operations

Series carry element-wise arithmetic (Add, Sub, Mul, Div, Mod, Exp, Neg), comparison (Eq, Ne, Lt, Le, Gt, Ge), and boolean (And, Or, Not) operators, plus Coalesce (fill nulls from another series or a scalar), Filter (by a []bool / []int or a Bools / Ints series), null-aware Group / SubGroup and Sort / SortRev, and Map, Slice, TakeIndices, Cast, Append.

Operators propagate NA: an operation with an NA operand yields NA, so x + NA is NA, as in SQL. Coalesce is the deliberate exception, since it exists to replace nulls.

DataFrame

Operation Status Notes
Select ✅ exact names; SelectMatching for regex
Filter ✅ by a Bools series
GroupBy + Agg ✅ null-aware group keys
Join ✅ inner / left / right / outer, null-aware
OrderBy ✅ multi-key, ascending / descending
Slice / TakeIndices ✅
Pivot (longer/wider) 🚧 in progress on dev-pivot
Map ❌ planned
Stack / Append ❌ planned

Select takes exact column names, in the order given; naming a column that does not exist is an error rather than a silently missing column. Pattern selection lives in SelectMatching, which takes regular expressions matched unanchored — so SelectMatching("Car") also selects CarOrigin.

Join output row order is part of the contract: rows follow the left frame's row order (a right join follows the right frame), a row with several matches produces consecutive rows ordered by the other frame, and an outer join puts the unmatched right rows last.

Aggregations (via Agg): Count, Sum, Mean, Min, Max, Std, Variance, Median, Quantile, Any, and All are all supported, with three deliberate behaviors:

  • Results are sorted by group key (ascending, nulls last).
  • Null values (NAs) are skipped by default; RemoveNAs(false) makes NAs propagate — a single NA in a group's value column makes that group's result NA for every aggregate (holistic and reducible alike), surfaced as NaN for the numeric aggregates and as null for Any/All.
  • Any / All return a Bools column, not Float64s.

Std/Variance take a WithDDoF option (default population, ddof=0); Median/Quantile take a WithInterpolation option.

Development

go test -short ./...     # fast unit tests (skips large local benchmark fixtures)
go test ./...            # full suite
go test -bench . ./...   # benchmarks (need local fixtures in testdata/)
go generate ./series/    # regenerate *_base.go / *_ops.go via the generators module

CI runs build, vet, gofmt, a generated-code freshness check, the test suite and the race detector on Linux and Windows.

Roadmap

Enchanter is pre-1.0. Releases are themed so each has a single focus, working toward a stable 1.0.

0.3.0 — measure & clean up. Deprecation cleanup (arrow.Record → RecordBatch), aggregation correctness fixes, CI + lint, and a measured decision on Arrow-native storage: not worth it — in pure Go, Arrow-backed columns only match plain Go slices, and arrow-go has no grouped-aggregation kernels, so Arrow stays an interop layer (Parquet, IPC, ecosystem handoff). See the storage measurement.

0.4.0 — make it fast. The performance headline, validated by a spike: single-pass, parallel hash aggregation for GroupBy().Agg() — on the h2oai-style Q1 sum-by-id benchmark (1e7 rows) it went from ~142ms to ~31ms, beating Polars' ~43ms. The remaining performance items moved to 0.6.0.

0.5.0 — stabilize for 1.0 (current; breaking changes, batched together):

  • Public API sealed: the internal series fields (Data_, NullMask_, IsNullable_, ...) are unexported. Data() and the typed accessors (Float64s(), Bools(), ...) are documented views of the backing storage; Strings gains Interned() for the raw pool pointers and every type gains PackedNullMask() for engines that walk the bit-packed mask. FilterIntSlice and its bounds-check toggle leave the interface; TakeIndices is the validated public path.
  • Broaden test coverage: join order and null-key contracts, dataframe Slice / TakeIndices / SelectAt (fixing SelectAt, whose inverted bounds check made it always fail), a CSV write/read round trip, the data-view contracts, and NA propagation across operators and operand orders (fixing x - NA, which errored while NA - x was NA).
  • Select split into an exact-name Select and a pattern-matching SelectMatching, so a plain column name can no longer silently over-select ("Car" used to also match "CarOrigin").
  • SAS7BDAT: decided — reading is delegated to kshedden/datareader (BSD-3-Clause) rather than finishing the bespoke parser, verified against that project's reference data; there is no write path (use XPT to hand data to SAS).
  • Internal naming: the legacy __gdl_ / leading-double-underscore helper names (Gandalff-era, pre-rename) are retired for idiomatic Go unexported names.
  • Generated *_ops.go readability: the length×nullability if nesting is flattened — nullability is resolved at run time by a shared null-mask helper and the length cases are a flat switch — shrinking the emitted operator code from ~44k to ~13k lines with byte-identical behavior.
  • Coalesce on every series type: fill null elements from another series or a scalar; a result element is null only when both operands are null there.
  • API reshape from the interface review: the one-implementation DataFrame interface is collapsed into the concrete type (NewDataFrame); IsError/GetError/IsErrored become the standard Err() error; the Get accessor prefix is dropped (Context, Partition, NullMask, ColIndex); IsSorted becomes SortOrder; the sort internals (Less, Equal, Swap) leave the Series interface; Not works on every series type; the join constants are JoinInner / JoinLeft / JoinRight / JoinOuter on JoinType; the undocumented variadic Take is replaced by Slice(start, end) and TakeIndices(indices); the column accessors are Col, ColAt, NameAt, ColIndex; reading starts from package-level constructors (dataframe.ReadCsv(ctx), ReadParquet, ...) and writing from WriteCsv, WriteParquet, ..., with the builder types exported; the dead SeriesNumeric interface is deleted; NA propagation now holds for every operator (String + NA no longer concatenates and Bool OR NA no longer copies values — both yield NA); the join row order is deterministic and documented.

0.6.0 — make it faster (the performance items deferred from 0.4.0 and 0.5.0, every one benchmark-gated):

  • Fused combinators for element-wise op chains (no intermediate arrays).
  • Benchmark-regression tracking so the gains don't rot.
  • Column-major / tiled aggregation loop experiment (#20): hoist the per-aggregator dispatch out of the row loop; needs a multi-aggregate benchmark first, and lands only if wide queries win with no single-aggregate regression.
  • Generics to collapse the generated per-type code — spike-gated (only if it measurably shrinks the code without regressing speed).

1.0 — commit to the stable API. The gate: the mutation and ownership contract of every Series method written down and tested.

Parking lot (unversioned, picked up as they fit): dictionary-encoded (factor) strings; pivot longer/wider (in progress on dev-pivot); custom aggregators (archived on archive/dev-fix-agg); stricter CSV type guessing (archived on archive/dev-0.1.3); chunked series; SPSS reader/writer; JSON-by-records; configurable time format; Set(i []int, …) and Slice(i []int); url resolvers and format options on the I/O builders; PrettyPrint and filtering-interface polish; memory-optimized Bool / uint64 null mask.

Dependencies

Built with:

License

See LICENSE.

Documentation

Overview

Package enchanter is a data-wrangling library for Go, in the spirit of pandas and Polars in Python and dplyr in R.

The entry point is a Context (see NewContext), which carries the string interning pool and the Apache Arrow memory allocator shared by every series and dataframe created from it: 

ctx := enchanter.NewContext()
df := dataframe.ReadCsv(ctx).
	SetReader(r).Read().
	GroupBy("department").
	Agg(dataframe.Mean("salary")).
	Run()

The functionality lives in the subpackages:

  • series: typed, nullable columns (Bool, Int, Int64, Float64, String, Time, Duration) with filtering, grouping, sorting and arithmetic.
  • dataframe: the DataFrame type — select, filter, group by, join, sort, aggregate — plus Apache Arrow record conversion.
  • io: readers and writers for CSV, XLSX, XPT (SAS), JSON, HTML, Markdown, Parquet and Arrow IPC, plus a read-only SAS7BDAT reader.
  • arrowutil: conversion helpers between enchanter null masks and Arrow validity bitmaps.
  • meta: the type system shared by the packages above.

Index

Constants

View Source 
const (
	// The default capacity of a series.
	DEFAULT_SERIES_INITIAL_CAPACITY = 10

	// The default capacity of a hash map.
	DEFAULT_HASH_MAP_INITIAL_CAPACITY = 1024

	// The default capacity of a dense map array.
	DEFAULT_DENSE_MAP_ARRAY_INITIAL_CAPACITY = 64

	// Number of threads to use for parallel operations.
	THREADS_NUMBER = 16

	// Minimum number of elements to use parallel operations.
	MINIMUM_PARALLEL_SIZE_1 = 16_384
	MINIMUM_PARALLEL_SIZE_2 = 131_072

	HASH_MAGIC_NUMBER      = int64(0xa8f4979b77e3f93)
	HASH_MAGIC_NUMBER_NULL = int64(0x7fff4979b77e3f93)
	HASH_NULL_KEY          = int64(0x7ff8000000000001)

	INF_TEXT         = "Inf"
	NA_TEXT          = "Na"
	EOL              = "\n"
	QUOTE            = "\""
	BOOL_TRUE_TEXT   = "true"
	BOOL_FALSE_TEXT  = "false"
	DATE_FORMAT      = "2006-01-02"
	DATE_TIME_FORMAT = "2006-01-02 15:04:05"

	CSV_READER_DEFAULT_DELIMITER           = ','
	CSV_READER_DEFAULT_HEADER              = true
	CSV_READER_DEFAULT_GUESS_DATA_TYPE_LEN = 1000

	XLSX_READER_DEFAULT_GUESS_DATA_TYPE_LEN = 1000
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Context 

type Context struct {
	// StringPool is a pool of strings that are used by the series.
	// This is used to reduce the number of allocations and to allow for fast comparisons.
	StringPool *StringPool

	// Allocator is the Arrow memory allocator used for building Arrow arrays.
	// Defaults to memory.DefaultAllocator (backed by Go GC, no manual Retain/Release needed).
	// Custom allocators must also be GC-backed (e.g. a CheckedAllocator wrapping a
	// GoAllocator): the library releases its internal Arrow references
	// deterministically, but Release on values returned to callers
	// (Series.ArrowArray, DataFrame.ToArrowRecord) is optional.
	Allocator memory.Allocator
	// contains filtered or unexported fields
}

func NewContext 

func NewContext() *Context

func (*Context) GetDateTimeFormat 

func (ctx *Context) GetDateTimeFormat() string

func (*Context) GetEol 

func (ctx *Context) GetEol() string

func (*Context) GetNaText 

func (ctx *Context) GetNaText() string

func (*Context) GetQuote 

func (ctx *Context) GetQuote() string

func (*Context) GetThreadsNumber 

func (ctx *Context) GetThreadsNumber() int

func (*Context) SetDateTimeFormat 

func (ctx *Context) SetDateTimeFormat(s string) *Context

func (*Context) SetEol 

func (ctx *Context) SetEol(s string) *Context

func (*Context) SetNaText 

func (ctx *Context) SetNaText(s string) *Context

func (*Context) SetQuote 

func (ctx *Context) SetQuote(s string) *Context

func (*Context) SetThreadsNumber 

func (ctx *Context) SetThreadsNumber(n int) *Context

type MapFunc 

type MapFunc func(v any) any

type MapFuncNull 

type MapFuncNull func(v any, isNull bool) (any, bool)

type NullableBool 

type NullableBool struct {
	Valid bool
	Value bool
}

type NullableDuration 

type NullableDuration struct {
	Valid bool
	Value time.Duration
}

type NullableFloat32 

type NullableFloat32 struct {
	Valid bool
	Value float32
}

type NullableFloat64 

type NullableFloat64 struct {
	Valid bool
	Value float64
}

type NullableInt 

type NullableInt struct {
	Valid bool
	Value int
}

type NullableInt8 

type NullableInt8 struct {
	Valid bool
	Value int8
}

type NullableInt16 

type NullableInt16 struct {
	Valid bool
	Value int16
}

type NullableInt32 

type NullableInt32 struct {
	Valid bool
	Value int32
}

type NullableInt64 

type NullableInt64 struct {
	Valid bool
	Value int64
}

type NullableString 

type NullableString struct {
	Valid bool
	Value string
}

type NullableTime 

type NullableTime struct {
	Valid bool
	Value time.Time
}

type SeriesSortOrder 

type SeriesSortOrder int16
const (
	// The series is not sorted.
	SORTED_NONE SeriesSortOrder = iota
	// The series is sorted in ascending order.
	SORTED_ASC
	// The series is sorted in descending order.
	SORTED_DESC
)

type StringPool 

type StringPool struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

StringPool is a pool of strings that are used by the series. This is used to reduce the number of allocations and to allow for fast comparisons.

func NewStringPool 

func NewStringPool() *StringPool

func (*StringPool) Get 

func (sp *StringPool) Get(s string) *string

Get returns the address of the string if it exists in the pool, otherwise nil.

func (*StringPool) Len 

func (sp *StringPool) Len() int

func (*StringPool) Put 

func (sp *StringPool) Put(s string) *string

Put returns the address of the string if it exists in the pool, otherwise it adds it to the pool and returns its address.

func (*StringPool) PutSync 

func (sp *StringPool) PutSync(s string) *string

PutSync returns the address of the string if it exists in the pool, otherwise it adds it to the pool and returns its address. This version is thread-safe.

func (*StringPool) SetNaText 

func (sp *StringPool) SetNaText(s string) *StringPool

func (*StringPool) ToString 

func (sp *StringPool) ToString() string

Source Files

View all Source files

Directories

Path Synopsis
Package arrowutil provides conversion utilities between enchanter's null mask convention (bit-set=null) and Arrow's validity bitmap convention (bit-set=valid).
 Package arrowutil provides conversion utilities between enchanter's null mask convention (bit-set=null) and Arrow's validity bitmap convention (bit-set=valid).
Package dataframe implements the DataFrame: an ordered collection of named, equally sized series with relational-style operations.
 Package dataframe implements the DataFrame: an ordered collection of named, equally sized series with relational-style operations.
Package io implements the file readers and writers behind the dataframe I/O chains: CSV, XLSX, XPT (SAS transport, versions 5-9), JSON, HTML, Markdown, Parquet, Arrow IPC (Feather) and SAS7BDAT (read-only, via kshedden/datareader).
 Package io implements the file readers and writers behind the dataframe I/O chains: CSV, XLSX, XPT (SAS transport, versions 5-9), JSON, HTML, Markdown, Parquet, Arrow IPC (Feather) and SAS7BDAT (read-only, via kshedden/datareader).
Package meta defines the type system shared across enchanter: the BaseType enumeration for series element types, type cardinality, and the operation metadata used by the code generator.
 Package meta defines the type system shared across enchanter: the BaseType enumeration for series element types, type cardinality, and the operation metadata used by the code generator.
Package series implements typed, nullable columns — the building block of dataframes.
 Package series implements typed, nullable columns — the building block of dataframes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.