vortex

README

vortex-go

A native Go reader for Vortex columnar files. It reads from io.ReaderAt and returns Apache Arrow record batches without cgo or a Rust runtime.

This is an independent implementation derived from the Rust Vortex format, schemas, and decoding algorithms. Development targets Rust Vortex revision 46a8d39c032b1e9f8ae28a13efe8abc9ebad556f; compatibility with arbitrary writer versions is not implied. The implementation is under active development.

input, err := os.Open("data.vortex")
if err != nil { return err }
defer input.Close()
info, err := input.Stat()
if err != nil { return err }
file, err := vortex.Open(ctx, input, info.Size())
if err != nil { return err }
defer file.Close()
reader, err := file.NewRecordReader(ctx, vortex.ScanOptions{
    Columns: []string{"id", "name"},
})
if err != nil { return err }
defer reader.Release()
for reader.Next() {
    record := reader.RecordBatch()
    // Consume record here. Call Retain to keep it past Next or Release,
    // then release that retained reference when finished.
    _ = record
}
return reader.Err()

Import the package as github.com/mprammer/vortex-go (package name vortex). The example also uses os and a caller-provided context ctx.

Reading and ownership

Open reads metadata and exposes Schema and NumRows. Scans load projected columns on demand and align batches by physical row ranges. A nil column list selects every column; an empty list requests row counts without value columns. The caller owns the source and must keep it open while scans are active.

Each record reader implements Arrow's array.RecordReader. RowOffset is the original physical start of its current contiguous batch, including after pruning. This lets consumers apply positional deletes correctly.

ScanOptions.Predicates provides conservative integer min/max zone pruning. Predicates are ANDed. Pruning can retain nonmatching rows: callers must evaluate their residual predicate. Unknown or unusable optional statistics retain data.

The initial scope is scalar columns: booleans, numeric primitives, UTF-8, binary, dates in days, and decimal values up to 128 bits. Nested tables and a writer are outside this implementation's scope. Unsupported representations return errors.

Development

Go 1.25 or newer is required.

CGO_ENABLED=0 go test ./...
go test -race ./...

Rust-produced fixtures check exact values and nulls, encoding boundaries, and file-layout behavior. The reference fixture generators live in dev/fixtures. testdata/rust/README.md records the upstream writer pin and the generator for each fixture family. The module's Go tests require no Rust installation.

Apache-2.0. See NOTICE for upstream algorithm and schema attribution.

Documentation

Overview

Package vortex reads Vortex files using native Go decoders.

Index

• Variables
• type CompareOp
• type File
  • func Open(ctx context.Context, source io.ReaderAt, size int64) (file *File, err error)
  • func (f *File) Close() error
  • func (f *File) NewRecordReader(ctx context.Context, opts ScanOptions) (*RecordReader, error)
  • func (f *File) NumRows() int64
  • func (f *File) Schema() *arrow.Schema
• type Predicate
• type RecordReader
  • func (r *RecordReader) Err() error
  • func (r *RecordReader) Next() (ok bool)
  • func (r *RecordReader) Record() arrow.RecordBatch
  • func (r *RecordReader) RecordBatch() arrow.RecordBatch
  • func (r *RecordReader) Release()
  • func (r *RecordReader) Retain()
  • func (r *RecordReader) RowOffset() int64
  • func (r *RecordReader) Schema() *arrow.Schema
• type ScanOptions

Variables

var (
	ErrInvalid     = errors.New("invalid Vortex file")
	ErrUnsupported = errors.New("unsupported Vortex feature")
)

Types

type CompareOp

type CompareOp uint8

CompareOp identifies an integer comparison used for conservative pruning.

const (
	OpEQ CompareOp = iota
	OpNEQ
	OpGT
	OpGTE
	OpLT
	OpLTE
)

type File

type File struct {
	// contains filtered or unexported fields
}

File holds immutable file metadata. It does not own or close its ReaderAt. Concurrent record readers are supported when the source supports concurrent reads. Individual metadata and data segments are limited to 64 MiB and 256 MiB, respectively. Every source read is at most 64 KiB.

func Open

func Open(ctx context.Context, source io.ReaderAt, size int64) (file *File, err error)

Open reads the file header and footer metadata, without decoding data arrays.

func (*File) Close

func (f *File) Close() error

Close leaves the caller-owned source open. RecordReader.Release frees scan buffers.

func (*File) NewRecordReader

func (f *File) NewRecordReader(ctx context.Context, opts ScanOptions) (*RecordReader, error)

func (*File) NumRows

func (f *File) NumRows() int64

func (*File) Schema

func (f *File) Schema() *arrow.Schema

type Predicate

type Predicate struct {
	Column string
	Op     CompareOp
	Value  int64
}

Predicate excludes only row ranges whose statistics prove the comparison false. Consumers must still apply the exact predicate to the returned rows.

type RecordReader

type RecordReader struct {
	// contains filtered or unexported fields
}

RecordReader streams projected batches in physical row order. Next and accessors must not run concurrently. Retain and Release follow Arrow ownership semantics. The current batch remains valid until Next or Release unless the caller retains it.

func (*RecordReader) Err

func (r *RecordReader) Err() error

func (*RecordReader) Next

func (r *RecordReader) Next() (ok bool)

func (*RecordReader) Record

func (r *RecordReader) Record() arrow.RecordBatch

Record is the deprecated Arrow compatibility accessor; use RecordBatch.

func (*RecordReader) RecordBatch

func (r *RecordReader) RecordBatch() arrow.RecordBatch

func (*RecordReader) Release

func (r *RecordReader) Release()

func (*RecordReader) Retain

func (r *RecordReader) Retain()

func (*RecordReader) RowOffset

func (r *RecordReader) RowOffset() int64

func (*RecordReader) Schema

func (r *RecordReader) Schema() *arrow.Schema

type ScanOptions

type ScanOptions struct {
	Columns    []string
	BatchSize  int
	Predicates []Predicate
}

ScanOptions configures a scan. Predicates are ANDed. A nil Columns selects all columns; an empty non-nil slice selects none. BatchSize defaults to 65536.