enchanter

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 12, 2026 License: MIT Imports: 5 Imported by: 0

README ยถ

Enchanter ๐Ÿง™โ€โ™‚๏ธ

Go Reference CI

Formerly known as aargh โ€” renamed in v0.2.0. The v0.1.x releases remain available under the old module path.

Enchanter is a data wrangling library in pure Go, in the spirit of Pandas and Polars in Python and dplyr in R: typed, nullable columns (series) composed into DataFrames with select / filter / group by / join / sort / aggregate pipelines. Series and DataFrames interoperate with Apache Arrow, which also powers the Parquet and Arrow IPC readers and writers.

Enchanter is a work in progress and the API is not stable yet.

Install

go get github.com/caerbannogwhite/enchanter

Requires Go 1.26+. Pure Go, no cgo.

Quick start

package main

import (
	"strings"

	"github.com/caerbannogwhite/enchanter"
	"github.com/caerbannogwhite/enchanter/dataframe"
)

func main() {
	data1 := `
name,age,weight,junior,department,salary band
Alice C,29,75.0,F,HR,4
John Doe,30,80.5,true,IT,2
Bob,31,85.0,F,IT,4
Jane H,25,60.0,false,IT,4
Mary,28,70.0,false,IT,3
Oliver,32,90.0,true,HR,1
Ursula,27,65.0,f,Business,4
Charlie,33,60.0,t,Business,2
Megan,26,55.0,F,IT,3`

	dataframe.NewBaseDataFrame(enchanter.NewContext()).
		FromCsv().
		SetReader(strings.NewReader(data1)).
		Read().
		Select("department", "age", "weight", "junior").
		GroupBy("department").
		Agg(dataframe.Min("age"), dataframe.Max("weight"), dataframe.Mean("junior"), dataframe.Count()).
		Run().
		PPrint(dataframe.NewPPrintParams().SetUseLipGloss(true))
}

//   BaseDataFrame: 3 rows, 5 columns
// โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
// โ”‚ department โ”‚ min(age) โ”‚ max(weight) โ”‚ mean(junior) โ”‚ n     โ”‚
// โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
// โ”‚ String     โ”‚ Float64  โ”‚ Float64     โ”‚ Float64      โ”‚ Int64 โ”‚
// โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
// โ”‚ HR         โ”‚    29.00 โ”‚       90.00 โ”‚       0.5000 โ”‚ 2.000 โ”‚
// โ”‚ IT         โ”‚    25.00 โ”‚       85.00 โ”‚       0.2000 โ”‚ 5.000 โ”‚
// โ”‚ Business   โ”‚    27.00 โ”‚       65.00 โ”‚       0.5000 โ”‚ 2.000 โ”‚
// โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

More runnable examples live in examples and in the package documentation on pkg.go.dev.

Reading and writing files

Format Read Write Notes
CSV โœ… โœ… delimiter, header, type guessing
Parquet โœ… โœ… via Apache Arrow; types + nulls preserved
Arrow IPC (Feather) โœ… โœ…
XPT (SAS transport) โœ… โœ… versions 5โ€“9
XLSX โœ… โœ…
JSON โœ… โœ… record-oriented
HTML โœ… โœ… tables
Markdown โœ… โœ… tables
SAS7BDAT ๐Ÿšง โ€” header parsing only; data reading planned

All readers and writers share the same builder style:

// Parquet round trip: types and nulls survive, unlike CSV.
err := df.ToParquet().SetPath("people.parquet").Write()

df2 := dataframe.NewBaseDataFrame(ctx).
	FromParquet().
	SetPath("people.parquet").
	Read()

See examples/parquet for a runnable version.

Apache Arrow interop

Every series can produce an Arrow array and every DataFrame an Arrow record batch, which makes Enchanter data exchangeable with anything that speaks Arrow (DuckDB, Polars, pandas, DataFusion, Spark, ...):

rec := df.ToArrowRecord() // freshly built, owned by the record
defer rec.Release()       // optional under the default GC-backed allocator

df2 := dataframe.NewBaseDataFrameFromArrowRecord(rec, ctx)

Conversion notes:

  • Enchanter null masks map onto Arrow validity bitmaps in both directions.
  • series.ArrowArrayToSeries accepts more Arrow types than Enchanter has series types (all integer widths, Float32, Date32/64, every timestamp unit, LargeString, ...) and funnels them onto the native types: integers normalize to int64, timestamps to nanoseconds.
  • Conversions are materialized copies โ€” no reference to the source Arrow memory is retained, so you may release inputs immediately.
  • Releasing values returned to you is optional under GC-backed allocators (the default); see Context.Allocator for the exact ownership contract.

Data types

Seven nullable, columnar series types. Nulls live in a separate bit-packed mask, so a column with no nulls carries no null overhead.

Series Go element type
Bools bool
Ints int
Int64s int64
Float64s float64
Strings *string (interned via a shared pool)
Times time.Time
Durations time.Duration

Not implemented (would be added on demand): narrower integers (Int8/16/32), Float32, complex numbers, and a bit-packed memory-optimized Bool.

Operations

Series carry element-wise arithmetic (Add, Sub, Mul, Div, Mod, Exp, Neg), comparison (Eq, Ne, Lt, Le, Gt, Ge), and boolean (And, Or, Not) operators, plus Filter (by a []bool / []int or a Bools / Ints series), null-aware Group / SubGroup and Sort / SortRev, and Map, Take, Cast, Append.

DataFrame

Operation Status Notes
Select โœ…
Filter โœ… by a Bools series
GroupBy + Agg โœ… null-aware group keys
Join โœ… inner / left / right / outer, null-aware
OrderBy โœ… multi-key, ascending / descending
Take โœ…
Pivot (longer/wider) ๐Ÿšง in progress on dev-pivot
Map โŒ planned
Stack / Append โŒ planned

Aggregations (via Agg): Count, Sum, Mean, Min, Max, and StdDev (population) are supported; Median, Variance, and Quantile are planned.

Development

go test -short ./...     # fast unit tests (skips large local benchmark fixtures)
go test ./...            # full suite
go test -bench . ./...   # benchmarks (need local fixtures in testdata/)
go generate ./series/    # regenerate *_base.go / *_ops.go via the generators module

CI runs build, vet, gofmt, a generated-code freshness check, the test suite and the race detector on Linux and Windows.

Roadmap

Enchanter is pre-1.0. Releases are themed so each has a single focus, working toward a stable 1.0.

0.3.0 โ€” measure & clean up (current). Deprecation cleanup (arrow.Record โ†’ RecordBatch), aggregation correctness fixes, CI + lint, and a measured decision on Arrow-native storage: not worth it โ€” in pure Go, Arrow-backed columns only match plain Go slices, and arrow-go has no grouped-aggregation kernels, so Arrow stays an interop layer (Parquet, IPC, ecosystem handoff). See the storage measurement.

0.4.0 โ€” make it fast. The performance headline, validated by a spike:

  • Single-pass, parallel hash aggregation for GroupBy().Agg() โ€” ~10โ€“17ร— over today's groupby, and class-leading on the h2oai Q1 benchmark.
  • Fused combinators for element-wise op chains (no intermediate arrays).
  • Benchmark-regression tracking so the gains don't rot.

0.5.0 โ€” stabilize for 1.0 (breaking changes, batched together):

  • Public API: hide internal fields (Data_, NullMask_), standardize the reader constructors.
  • Generics to collapse the generated per-type code โ€” spike-gated (only if it measurably shrinks the code without regressing speed).
  • Broaden test coverage; decide SAS7BDAT (format notes) โ€” finish the data path or drop it.

1.0 โ€” commit to the stable API.

Parking lot (unversioned, picked up as they fit): dictionary-encoded (factor) strings; pivot longer/wider (in progress on dev-pivot); custom aggregators (archived on archive/dev-fix-agg); stricter CSV type guessing (archived on archive/dev-0.1.3); chunked series; SPSS reader/writer; JSON-by-records; Median / Variance / Quantile; configurable time format; Set(i []int, โ€ฆ) and Slice(i []int); url resolvers and format options on the I/O builders; PrettyPrint and filtering-interface polish; memory-optimized Bool / uint64 null mask.

Dependencies

Built with:

License

See LICENSE.

Documentation ยถ

Overview ยถ

Package enchanter is a data-wrangling library for Go, in the spirit of pandas and Polars in Python and dplyr in R.

The entry point is a Context (see NewContext), which carries the string interning pool and the Apache Arrow memory allocator shared by every series and dataframe created from it:

ctx := enchanter.NewContext()
df := dataframe.NewBaseDataFrame(ctx).
	FromCsv().SetReader(r).Read().
	GroupBy("department").
	Agg(dataframe.Mean("salary")).
	Run()

The functionality lives in the subpackages:

  • series: typed, nullable columns (Bool, Int, Int64, Float64, String, Time, Duration) with filtering, grouping, sorting and arithmetic.
  • dataframe: the DataFrame interface โ€” select, filter, group by, join, sort, aggregate โ€” plus Apache Arrow record conversion.
  • io: readers and writers for CSV, XLSX, XPT (SAS), JSON, HTML, Markdown, Parquet and Arrow IPC.
  • arrowutil: conversion helpers between enchanter null masks and Arrow validity bitmaps.
  • meta: the type system shared by the packages above.

Index ยถ

Constants ยถ

View Source
const (
	// The default capacity of a series.
	DEFAULT_SERIES_INITIAL_CAPACITY = 10

	// The default capacity of a hash map.
	DEFAULT_HASH_MAP_INITIAL_CAPACITY = 1024

	// The default capacity of a dense map array.
	DEFAULT_DENSE_MAP_ARRAY_INITIAL_CAPACITY = 64

	// Number of threads to use for parallel operations.
	THREADS_NUMBER = 16

	// Minimum number of elements to use parallel operations.
	MINIMUM_PARALLEL_SIZE_1 = 16_384
	MINIMUM_PARALLEL_SIZE_2 = 131_072

	HASH_MAGIC_NUMBER      = int64(0xa8f4979b77e3f93)
	HASH_MAGIC_NUMBER_NULL = int64(0x7fff4979b77e3f93)
	HASH_NULL_KEY          = int64(0x7ff8000000000001)

	INF_TEXT         = "Inf"
	NA_TEXT          = "Na"
	EOL              = "\n"
	QUOTE            = "\""
	BOOL_TRUE_TEXT   = "true"
	BOOL_FALSE_TEXT  = "false"
	DATE_FORMAT      = "2006-01-02"
	DATE_TIME_FORMAT = "2006-01-02 15:04:05"

	CSV_READER_DEFAULT_DELIMITER           = ','
	CSV_READER_DEFAULT_HEADER              = true
	CSV_READER_DEFAULT_GUESS_DATA_TYPE_LEN = 1000

	XLSX_READER_DEFAULT_GUESS_DATA_TYPE_LEN = 1000
)

Variables ยถ

This section is empty.

Functions ยถ

This section is empty.

Types ยถ

type Context ยถ

type Context struct {
	// StringPool is a pool of strings that are used by the series.
	// This is used to reduce the number of allocations and to allow for fast comparisons.
	StringPool *StringPool

	// Allocator is the Arrow memory allocator used for building Arrow arrays.
	// Defaults to memory.DefaultAllocator (backed by Go GC, no manual Retain/Release needed).
	// Custom allocators must also be GC-backed (e.g. a CheckedAllocator wrapping a
	// GoAllocator): the library releases its internal Arrow references
	// deterministically, but Release on values returned to callers
	// (Series.ArrowArray, DataFrame.ToArrowRecord) is optional.
	Allocator memory.Allocator
	// contains filtered or unexported fields
}

func NewContext ยถ

func NewContext() *Context

func (*Context) GetDateTimeFormat ยถ

func (ctx *Context) GetDateTimeFormat() string

func (*Context) GetEol ยถ

func (ctx *Context) GetEol() string

func (*Context) GetNaText ยถ

func (ctx *Context) GetNaText() string

func (*Context) GetQuote ยถ

func (ctx *Context) GetQuote() string

func (*Context) GetThreadsNumber ยถ

func (ctx *Context) GetThreadsNumber() int

func (*Context) SetDateTimeFormat ยถ

func (ctx *Context) SetDateTimeFormat(s string) *Context

func (*Context) SetEol ยถ

func (ctx *Context) SetEol(s string) *Context

func (*Context) SetNaText ยถ

func (ctx *Context) SetNaText(s string) *Context

func (*Context) SetQuote ยถ

func (ctx *Context) SetQuote(s string) *Context

func (*Context) SetThreadsNumber ยถ

func (ctx *Context) SetThreadsNumber(n int) *Context

type MapFunc ยถ

type MapFunc func(v any) any

type MapFuncNull ยถ

type MapFuncNull func(v any, isNull bool) (any, bool)

type NullableBool ยถ

type NullableBool struct {
	Valid bool
	Value bool
}

type NullableDuration ยถ

type NullableDuration struct {
	Valid bool
	Value time.Duration
}

type NullableFloat32 ยถ

type NullableFloat32 struct {
	Valid bool
	Value float32
}

type NullableFloat64 ยถ

type NullableFloat64 struct {
	Valid bool
	Value float64
}

type NullableInt ยถ

type NullableInt struct {
	Valid bool
	Value int
}

type NullableInt8 ยถ

type NullableInt8 struct {
	Valid bool
	Value int8
}

type NullableInt16 ยถ

type NullableInt16 struct {
	Valid bool
	Value int16
}

type NullableInt32 ยถ

type NullableInt32 struct {
	Valid bool
	Value int32
}

type NullableInt64 ยถ

type NullableInt64 struct {
	Valid bool
	Value int64
}

type NullableString ยถ

type NullableString struct {
	Valid bool
	Value string
}

type NullableTime ยถ

type NullableTime struct {
	Valid bool
	Value time.Time
}

type SeriesSortOrder ยถ

type SeriesSortOrder int16
const (
	// The series is not sorted.
	SORTED_NONE SeriesSortOrder = iota
	// The series is sorted in ascending order.
	SORTED_ASC
	// The series is sorted in descending order.
	SORTED_DESC
)

type StringPool ยถ

type StringPool struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

StringPool is a pool of strings that are used by the series. This is used to reduce the number of allocations and to allow for fast comparisons.

func NewStringPool ยถ

func NewStringPool() *StringPool

func (*StringPool) Get ยถ

func (sp *StringPool) Get(s string) *string

Get returns the address of the string if it exists in the pool, otherwise nil.

func (*StringPool) Len ยถ

func (sp *StringPool) Len() int

func (*StringPool) Put ยถ

func (sp *StringPool) Put(s string) *string

Put returns the address of the string if it exists in the pool, otherwise it adds it to the pool and returns its address.

func (*StringPool) PutSync ยถ

func (sp *StringPool) PutSync(s string) *string

PutSync returns the address of the string if it exists in the pool, otherwise it adds it to the pool and returns its address. This version is thread-safe.

func (*StringPool) SetNaText ยถ

func (sp *StringPool) SetNaText(s string) *StringPool

func (*StringPool) ToString ยถ

func (sp *StringPool) ToString() string

Directories ยถ

Path Synopsis
Package arrowutil provides conversion utilities between enchanter's null mask convention (bit-set=null) and Arrow's validity bitmap convention (bit-set=valid).
Package arrowutil provides conversion utilities between enchanter's null mask convention (bit-set=null) and Arrow's validity bitmap convention (bit-set=valid).
Package dataframe implements the DataFrame: an ordered collection of named, equally sized series with relational-style operations.
Package dataframe implements the DataFrame: an ordered collection of named, equally sized series with relational-style operations.
Package io implements the file readers and writers behind the dataframe I/O chains: CSV, XLSX, XPT (SAS transport, versions 5-9), JSON, HTML, Markdown, Parquet and Arrow IPC (Feather).
Package io implements the file readers and writers behind the dataframe I/O chains: CSV, XLSX, XPT (SAS transport, versions 5-9), JSON, HTML, Markdown, Parquet and Arrow IPC (Feather).
Package meta defines the type system shared across enchanter: the BaseType enumeration for series element types, type cardinality, and the operation metadata used by the code generator.
Package meta defines the type system shared across enchanter: the BaseType enumeration for series element types, type cardinality, and the operation metadata used by the code generator.
Package series implements typed, nullable columns โ€” the building block of dataframes.
Package series implements typed, nullable columns โ€” the building block of dataframes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL