enchanter

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 6, 2026 License: MIT Imports: 5 Imported by: 0

README ยถ

Enchanter ๐Ÿง™โ€โ™‚๏ธ

Go Reference CI

Formerly known as aargh โ€” renamed in v0.2.0. The v0.1.x releases remain available under the old module path.

Enchanter is a data wrangling library in pure Go, in the spirit of Pandas and Polars in Python and dplyr in R: typed, nullable columns (series) composed into DataFrames with select / filter / group by / join / sort / aggregate pipelines. Series and DataFrames interoperate with Apache Arrow, which also powers the Parquet and Arrow IPC readers and writers.

Enchanter is a work in progress and the API is not stable yet.

Install

go get github.com/caerbannogwhite/enchanter

Requires Go 1.24+. Pure Go, no cgo.

Quick start

package main

import (
	"strings"

	"github.com/caerbannogwhite/enchanter"
	"github.com/caerbannogwhite/enchanter/dataframe"
)

func main() {
	data1 := `
name,age,weight,junior,department,salary band
Alice C,29,75.0,F,HR,4
John Doe,30,80.5,true,IT,2
Bob,31,85.0,F,IT,4
Jane H,25,60.0,false,IT,4
Mary,28,70.0,false,IT,3
Oliver,32,90.0,true,HR,1
Ursula,27,65.0,f,Business,4
Charlie,33,60.0,t,Business,2
Megan,26,55.0,F,IT,3`

	dataframe.NewBaseDataFrame(enchanter.NewContext()).
		FromCsv().
		SetReader(strings.NewReader(data1)).
		Read().
		Select("department", "age", "weight", "junior").
		GroupBy("department").
		Agg(dataframe.Min("age"), dataframe.Max("weight"), dataframe.Mean("junior"), dataframe.Count()).
		Run().
		PPrint(dataframe.NewPPrintParams().SetUseLipGloss(true))
}

//   BaseDataFrame: 3 rows, 5 columns
// โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
// โ”‚ department โ”‚ min(age) โ”‚ max(weight) โ”‚ mean(junior) โ”‚ n     โ”‚
// โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
// โ”‚ String     โ”‚ Float64  โ”‚ Float64     โ”‚ Float64      โ”‚ Int64 โ”‚
// โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
// โ”‚ HR         โ”‚    29.00 โ”‚       90.00 โ”‚       0.5000 โ”‚ 2.000 โ”‚
// โ”‚ IT         โ”‚    25.00 โ”‚       85.00 โ”‚       0.2000 โ”‚ 5.000 โ”‚
// โ”‚ Business   โ”‚    27.00 โ”‚       65.00 โ”‚       0.5000 โ”‚ 2.000 โ”‚
// โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

More runnable examples live in examples and in the package documentation on pkg.go.dev.

Reading and writing files

Format Read Write Notes
CSV โœ… โœ… delimiter, header, type guessing
Parquet โœ… โœ… via Apache Arrow; types + nulls preserved
Arrow IPC (Feather) โœ… โœ…
XPT (SAS transport) โœ… โœ… versions 5โ€“9
XLSX โœ… โœ…
JSON โœ… โœ… record-oriented
HTML โœ… โœ… tables
Markdown โœ… โœ… tables
SAS7BDAT ๐Ÿšง โ€” header parsing only; data reading planned

All readers and writers share the same builder style:

// Parquet round trip: types and nulls survive, unlike CSV.
err := df.ToParquet().SetPath("people.parquet").Write()

df2 := dataframe.NewBaseDataFrame(ctx).
	FromParquet().
	SetPath("people.parquet").
	Read()

See examples/parquet for a runnable version.

Apache Arrow interop

Every series can produce an Arrow array and every DataFrame an Arrow record batch, which makes Enchanter data exchangeable with anything that speaks Arrow (DuckDB, Polars, pandas, DataFusion, Spark, ...):

rec := df.ToArrowRecord() // freshly built, owned by the record
defer rec.Release()       // optional under the default GC-backed allocator

df2 := dataframe.NewBaseDataFrameFromArrowRecord(rec, ctx)

Conversion notes:

  • Enchanter null masks map onto Arrow validity bitmaps in both directions.
  • series.ArrowArrayToSeries accepts more Arrow types than Enchanter has series types (all integer widths, Float32, Date32/64, every timestamp unit, LargeString, ...) and funnels them onto the native types: integers normalize to int64, timestamps to nanoseconds.
  • Conversions are materialized copies โ€” no reference to the source Arrow memory is retained, so you may release inputs immediately.
  • Releasing values returned to you is optional under GC-backed allocators (the default); see Context.Allocator for the exact ownership contract.

Supported data types

The data types not checked are not yet supported, but might be in the future.

  • Bool
  • Bool (memory optimized, not fully implemented yet)
  • Int16
  • Int
  • Int64
  • Float32
  • Float64
  • Complex64
  • Complex128
  • String
  • Time
  • Duration

Supported operations for Series

  • Filter

    • filter by bool slice
    • filter by int slice
    • filter by bool series
    • filter by int series
  • Group

    • Group (with nulls)
    • SubGroup (with nulls)
  • Map

  • Sort

    • Sort (with nulls)
    • SortRev (with nulls)
  • Take

Supported operations for DataFrame

  • Agg

  • Filter

  • GroupBy

  • Join

    • Inner
    • Left
    • Right
    • Outer
    • Inner with nulls
    • Left with nulls
    • Right with nulls
    • Outer with nulls
  • Map

  • OrderBy

  • Select

  • Take

  • Pivot

  • Stack/Append

Supported stats functions

  • Count
  • Sum
  • Mean
  • Median
  • Min
  • Max
  • StdDev
  • Variance
  • Quantile

Development

go test -short ./...     # fast unit tests (skips large local benchmark fixtures)
go test ./...            # full suite
go test -bench . ./...   # benchmarks (need local fixtures in testdata/)
go generate ./series/    # regenerate *_base.go / *_ops.go via the generators module

CI runs build, vet, gofmt, a generated-code freshness check, the test suite and the race detector on Linux and Windows.

Roadmap

Next (0.3.0):

  • Route series operations through Arrow compute kernels (replacing the generated per-type loops).
  • Aggregations via Arrow compute (Sum, Min, Max, Mean).
  • Dictionary-encoded (factor) strings.
  • SAS7BDAT data reading (format notes).

Later:

  • Implement chunked series.
  • Implement SPSS reader and writer.
  • Improve filtering interface.
  • Improve dataframe PrettyPrint: add parameters, optimize data display, use lipgloss.
  • Times: set time format.
  • Implement Set(i []int, v []any) Series.
  • Add Slice(i []int) Series (using filter?).
  • Implement memory optimized Bool series with uint64.
  • Use uint64 for null mask.
  • Optimize XPT reader/writer with float32.
  • Add url resolver to each reader.
  • Add format option to each writer.
  • JSON reader by records.

Dependencies

Built with:

License

See LICENSE.

Documentation ยถ

Overview ยถ

Package enchanter is a data-wrangling library for Go, in the spirit of pandas and Polars in Python and dplyr in R.

The entry point is a Context (see NewContext), which carries the string interning pool and the Apache Arrow memory allocator shared by every series and dataframe created from it:

ctx := enchanter.NewContext()
df := dataframe.NewBaseDataFrame(ctx).
	FromCsv().SetReader(r).Read().
	GroupBy("department").
	Agg(dataframe.Mean("salary")).
	Run()

The functionality lives in the subpackages:

  • series: typed, nullable columns (Bool, Int, Int64, Float64, String, Time, Duration) with filtering, grouping, sorting and arithmetic.
  • dataframe: the DataFrame interface โ€” select, filter, group by, join, sort, aggregate โ€” plus Apache Arrow record conversion.
  • io: readers and writers for CSV, XLSX, XPT (SAS), JSON, HTML, Markdown, Parquet and Arrow IPC.
  • arrowutil: conversion helpers between enchanter null masks and Arrow validity bitmaps.
  • meta: the type system shared by the packages above.

Index ยถ

Constants ยถ

View Source
const (
	// The default capacity of a series.
	DEFAULT_SERIES_INITIAL_CAPACITY = 10

	// The default capacity of a hash map.
	DEFAULT_HASH_MAP_INITIAL_CAPACITY = 1024

	// The default capacity of a dense map array.
	DEFAULT_DENSE_MAP_ARRAY_INITIAL_CAPACITY = 64

	// Number of threads to use for parallel operations.
	THREADS_NUMBER = 16

	// Minimum number of elements to use parallel operations.
	MINIMUM_PARALLEL_SIZE_1 = 16_384
	MINIMUM_PARALLEL_SIZE_2 = 131_072

	HASH_MAGIC_NUMBER      = int64(0xa8f4979b77e3f93)
	HASH_MAGIC_NUMBER_NULL = int64(0x7fff4979b77e3f93)
	HASH_NULL_KEY          = int64(0x7ff8000000000001)

	INF_TEXT         = "Inf"
	NA_TEXT          = "Na"
	EOL              = "\n"
	QUOTE            = "\""
	BOOL_TRUE_TEXT   = "true"
	BOOL_FALSE_TEXT  = "false"
	DATE_FORMAT      = "2006-01-02"
	DATE_TIME_FORMAT = "2006-01-02 15:04:05"

	CSV_READER_DEFAULT_DELIMITER           = ','
	CSV_READER_DEFAULT_HEADER              = true
	CSV_READER_DEFAULT_GUESS_DATA_TYPE_LEN = 1000

	XLSX_READER_DEFAULT_GUESS_DATA_TYPE_LEN = 1000
)

Variables ยถ

This section is empty.

Functions ยถ

This section is empty.

Types ยถ

type Context ยถ

type Context struct {
	// StringPool is a pool of strings that are used by the series.
	// This is used to reduce the number of allocations and to allow for fast comparisons.
	StringPool *StringPool

	// Allocator is the Arrow memory allocator used for building Arrow arrays.
	// Defaults to memory.DefaultAllocator (backed by Go GC, no manual Retain/Release needed).
	// Custom allocators must also be GC-backed (e.g. a CheckedAllocator wrapping a
	// GoAllocator): the library releases its internal Arrow references
	// deterministically, but Release on values returned to callers
	// (Series.ArrowArray, DataFrame.ToArrowRecord) is optional.
	Allocator memory.Allocator
	// contains filtered or unexported fields
}

func NewContext ยถ

func NewContext() *Context

func (*Context) GetDateTimeFormat ยถ

func (ctx *Context) GetDateTimeFormat() string

func (*Context) GetEol ยถ

func (ctx *Context) GetEol() string

func (*Context) GetNaText ยถ

func (ctx *Context) GetNaText() string

func (*Context) GetQuote ยถ

func (ctx *Context) GetQuote() string

func (*Context) GetThreadsNumber ยถ

func (ctx *Context) GetThreadsNumber() int

func (*Context) SetDateTimeFormat ยถ

func (ctx *Context) SetDateTimeFormat(s string) *Context

func (*Context) SetEol ยถ

func (ctx *Context) SetEol(s string) *Context

func (*Context) SetNaText ยถ

func (ctx *Context) SetNaText(s string) *Context

func (*Context) SetQuote ยถ

func (ctx *Context) SetQuote(s string) *Context

func (*Context) SetThreadsNumber ยถ

func (ctx *Context) SetThreadsNumber(n int) *Context

type MapFunc ยถ

type MapFunc func(v any) any

type MapFuncNull ยถ

type MapFuncNull func(v any, isNull bool) (any, bool)

type NullableBool ยถ

type NullableBool struct {
	Valid bool
	Value bool
}

type NullableDuration ยถ

type NullableDuration struct {
	Valid bool
	Value time.Duration
}

type NullableFloat32 ยถ

type NullableFloat32 struct {
	Valid bool
	Value float32
}

type NullableFloat64 ยถ

type NullableFloat64 struct {
	Valid bool
	Value float64
}

type NullableInt ยถ

type NullableInt struct {
	Valid bool
	Value int
}

type NullableInt8 ยถ

type NullableInt8 struct {
	Valid bool
	Value int8
}

type NullableInt16 ยถ

type NullableInt16 struct {
	Valid bool
	Value int16
}

type NullableInt32 ยถ

type NullableInt32 struct {
	Valid bool
	Value int32
}

type NullableInt64 ยถ

type NullableInt64 struct {
	Valid bool
	Value int64
}

type NullableString ยถ

type NullableString struct {
	Valid bool
	Value string
}

type NullableTime ยถ

type NullableTime struct {
	Valid bool
	Value time.Time
}

type SeriesSortOrder ยถ

type SeriesSortOrder int16
const (
	// The series is not sorted.
	SORTED_NONE SeriesSortOrder = iota
	// The series is sorted in ascending order.
	SORTED_ASC
	// The series is sorted in descending order.
	SORTED_DESC
)

type StringPool ยถ

type StringPool struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

StringPool is a pool of strings that are used by the series. This is used to reduce the number of allocations and to allow for fast comparisons.

func NewStringPool ยถ

func NewStringPool() *StringPool

func (*StringPool) Get ยถ

func (sp *StringPool) Get(s string) *string

Get returns the address of the string if it exists in the pool, otherwise nil.

func (*StringPool) Len ยถ

func (sp *StringPool) Len() int

func (*StringPool) Put ยถ

func (sp *StringPool) Put(s string) *string

Put returns the address of the string if it exists in the pool, otherwise it adds it to the pool and returns its address.

func (*StringPool) PutSync ยถ

func (sp *StringPool) PutSync(s string) *string

PutSync returns the address of the string if it exists in the pool, otherwise it adds it to the pool and returns its address. This version is thread-safe.

func (*StringPool) SetNaText ยถ

func (sp *StringPool) SetNaText(s string) *StringPool

func (*StringPool) ToString ยถ

func (sp *StringPool) ToString() string

Directories ยถ

Path Synopsis
Package arrowutil provides conversion utilities between enchanter's null mask convention (bit-set=null) and Arrow's validity bitmap convention (bit-set=valid).
Package arrowutil provides conversion utilities between enchanter's null mask convention (bit-set=null) and Arrow's validity bitmap convention (bit-set=valid).
Package dataframe implements the DataFrame: an ordered collection of named, equally sized series with relational-style operations.
Package dataframe implements the DataFrame: an ordered collection of named, equally sized series with relational-style operations.
Package io implements the file readers and writers behind the dataframe I/O chains: CSV, XLSX, XPT (SAS transport, versions 5-9), JSON, HTML, Markdown, Parquet and Arrow IPC (Feather).
Package io implements the file readers and writers behind the dataframe I/O chains: CSV, XLSX, XPT (SAS transport, versions 5-9), JSON, HTML, Markdown, Parquet and Arrow IPC (Feather).
Package meta defines the type system shared across enchanter: the BaseType enumeration for series element types, type cardinality, and the operation metadata used by the code generator.
Package meta defines the type system shared across enchanter: the BaseType enumeration for series element types, type cardinality, and the operation metadata used by the code generator.
Package series implements typed, nullable columns โ€” the building block of dataframes.
Package series implements typed, nullable columns โ€” the building block of dataframes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL