parquet

package module
v0.0.0-...-275ee29 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 17, 2026 License: BSD-3-Clause Imports: 15 Imported by: 0

README

go-ruby-parquet/parquet

parquet — go-ruby-parquet

Docs License Go Coverage

A pure-Go (no cgo) reimplementation of Ruby's red-parquet gem — reading and writing Apache Parquet files as Apache Arrow tables. The real red-parquet gem binds the C++ library libparquet through GObject introspection over Apache Arrow, so it cannot ship inside a CGO-free static binary. This package mirrors red-parquet's observable Ruby surface — Parquet::ArrowFileReader, Parquet::ArrowFileWriter, the Parquet::Writer convenience, Table#save / Table.load format dispatch and the Parquet::*Error tree — on top of the Parquet reader/writer of github.com/apache/arrow-go/v18/parquet and its Arrow bridge parquet/pqarrow, the official pure-Go Apache Parquet implementation. It does not reimplement the Parquet format; it re-presents arrow-go's Parquet stack through Ruby's naming and semantics.

It pairs with go-ruby-arrow: the tables this library reads and writes are go-ruby-arrow Tables, so the two compose at the Ruby level — build a Table with go-ruby-arrow, persist it here, load it back, hand it to any go-ruby-arrow consumer unchanged. It is a sibling of go-ruby-marshal and the other go-ruby-* satellite libraries, and a backend for go-embedded-ruby.

Consumes, does not reinvent. The columnar decode/encode, row groups, compression codecs and Parquet metadata all come from arrow-go. The value this package adds is the faithful Ruby surface and the go-ruby-arrow Table interop, verified wire-compatible with arrow-go's canonical pqarrow.FileReader/writer in both directions.

Install

go get github.com/go-ruby-parquet/parquet

Usage

package main

import (
	"bytes"
	"fmt"

	arrow "github.com/go-ruby-arrow/arrow"
	parquet "github.com/go-ruby-parquet/parquet"
)

func main() {
	schema := arrow.NewSchema(
		arrow.NewField("id", arrow.Int64()),
		arrow.NewField("name", arrow.StringType()),
	)
	id, _ := arrow.NewArrayOf(arrow.Int64(), []any{int64(1), int64(2), nil})
	name, _ := arrow.NewArrayOf(arrow.StringType(), []any{"a", "b", "c"})
	table, _ := arrow.NewTable(schema, []*arrow.Array{id, name})

	// Write the Arrow table to Parquet (Zstd, 1000-row row groups).
	var buf bytes.Buffer
	_ = parquet.WriteTableTo(&buf, table,
		parquet.WithCompression(parquet.Zstd),
		parquet.WithRowGroupSize(1000))

	// Read it back as a go-ruby-arrow Table.
	back, _ := parquet.ReadTableBytes(buf.Bytes())
	fmt.Println(back.NumRows(), back.NumColumns()) // 3 2

	col, _ := back.Column("name")
	v, _ := col.Get(2)
	fmt.Println(v) // c
}

Ruby-to-Go mapping

Ruby (red-parquet) Go (this package)
Parquet::ArrowFileReader.new NewArrowFileReader (IO) / OpenArrowFileReader (path)
#read_table (*ArrowFileReader).ReadTable
#read_row_group(i) (*ArrowFileReader).ReadRowGroup
#n_rows / #n_row_groups NumRows / NumRowGroups
#schema (*ArrowFileReader).Schema
Parquet::ArrowFileWriter.new NewArrowFileWriter — Write, Close
Parquet::Writer.write(t, path) WriteTable / WriteTableTo
Arrow::Table#save("x.parquet") Save
Arrow::Table.load("x.parquet") Load
Parquet::WriterProperties WithCompression / WithRowGroupSize / WithDictionary
Parquet::Error tree *Error (Kind + RubyClass())

Compression symbols (:uncompressed / :snappy / :gzip / :zstd) map to the Compression constants and [ParseCompression].

Round-trip & wire compatibility

Every Arrow column type — Int8..Int64, UInt8..UInt64, Float32/Float64, Boolean, String, Timestamp, Date32, Decimal128, List and Struct, each with nulls — round-trips through Parquet with values and schema preserved, under every codec (uncompressed / snappy / gzip / zstd). Cross-library wire compatibility is verified, not asserted: a file written here is read back by arrow-go's canonical pqarrow.ReadTable, and a file written by arrow-go's canonical pqarrow.WriteTable is read here — both directions, entirely in memory (no network). Parquet is a little-endian on-disk format; arrow-go handles the byte swap on big-endian targets (s390x), so files round-trip identically across all six supported 64-bit arches.

Tests & coverage

The suite is deterministic and dependency-light (no libparquet, CGO=0): all-types/all-codecs round-trips, per-row-group reads, path and IO constructors, the full error tree, and the cross-checks against arrow-go's canonical reader/writer that pin wire compatibility.

COVERPKG=$(go list ./... | paste -sd, -)
go test -race -coverpkg="$COVERPKG" -coverprofile=cover.out ./...
go tool cover -func=cover.out | tail -1   # 100.0%

CGO-free, gofmt + go vet clean, and green across the six 64-bit Go targets (amd64, arm64, riscv64, loong64, ppc64le, s390x — the last big-endian) and three OSes (Linux, macOS, Windows).

Scope

This covers red-parquet's Arrow-table read/write path plus compression, row groups and dictionary encoding. It does not (yet) cover the lower-level column-chunk statistics DSL, Bloom filters, encryption, or the Dataset API; those are additive follow-ups on the same arrow-go foundation. See doc.go for the authoritative scope note.

License

BSD-3-Clause — see LICENSE. Copyright the go-ruby-parquet/parquet authors.

WebAssembly

Being pure Go (CGO=0), this library also compiles to WebAssembly — both GOOS=js GOARCH=wasm (browser / Node.js) and GOOS=wasip1 GOARCH=wasm (WASI). CI builds both targets on every push, alongside the six 64-bit native/qemu arches.

GOOS=js     GOARCH=wasm go build ./...   # browser / Node
GOOS=wasip1 GOARCH=wasm go build ./...   # WASI (wasmtime, wasmer, wasmedge, …)

Documentation

Overview

Package parquet is a pure-Go (CGO=0), MRI-faithful implementation of the Ruby red-parquet gem's core surface — reading and writing Apache Parquet files as Apache Arrow tables.

Relationship to upstream

The real red-parquet gem is a thin Ruby binding over the C++ library libparquet (via GObject introspection over Apache Arrow), so it cannot be shipped in a CGO-free static binary. This package mirrors red-parquet's observable Ruby surface — Parquet::ArrowFileReader, Parquet::ArrowFileWriter, the Parquet::Writer convenience, Table#save / Table.load format dispatch and the Parquet::*Error tree — on top of the Parquet reader/writer of github.com/apache/arrow-go/v18/parquet and its Arrow bridge github.com/apache/arrow-go/v18/parquet/pqarrow, the official pure-Go Apache Parquet implementation. It does not reimplement the Parquet format; it re-presents arrow-go's Parquet stack through Ruby's naming and semantics so it can back an embedded Ruby (go-embedded-ruby / rbgo) with no cgo.

Interoperability with go-ruby-arrow

The Arrow tables this package reads and writes are github.com/go-ruby-arrow/arrow Tables — the same type red-arrow is mapped to — so the two libraries compose at the Ruby level: build a Table with go-ruby-arrow, persist it here, load it back, and hand it to any go-ruby-arrow consumer unchanged.

Ruby-to-Go mapping 

Parquet::ArrowFileReader -> *ArrowFileReader (ReadTable/ReadRowGroup/NumRows/…)
Parquet::ArrowFileWriter -> *ArrowFileWriter (Write/Close)
Parquet::Writer.write     -> WriteTable / WriteTableTo / Save
Arrow::Table.load         -> Load / ReadTable
Arrow::Table#save         -> Save
Parquet::Error (tree)     -> *Error (Kind + RubyClass mapping)

Compression and encoding

The writer honours red-parquet's per-file compression symbols (:uncompressed / :snappy / :gzip / :zstd) via WithCompression, the row-group size via WithRowGroupSize, and dictionary encoding via WithDictionary. Every codec round-trips.

Wire compatibility

A Parquet file written here is read back by arrow-go's canonical pqarrow.FileReader, and a file written by arrow-go's canonical writer is read here — both directions are verified by the differential tests, not asserted. Parquet is a little-endian on-disk format; arrow-go handles the byte swap on big-endian targets (s390x), so the same files round-trip identically across all six supported 64-bit arches.

Index

Constants

View Source 
const DefaultRowGroupSize int64 = 1024 * 1024

DefaultRowGroupSize is the number of rows per row group used when no WithRowGroupSize option is given, matching arrow-go's default row-group length.

Variables

View Source 
var (
	// ErrType matches KindType errors, ErrIndex matches KindIndex, etc.
	ErrType           = &Error{Kind: KindType}
	ErrIndex          = &Error{Kind: KindIndex}
	ErrArgument       = &Error{Kind: KindArgument}
	ErrIO             = &Error{Kind: KindIO}
	ErrNotImplemented = &Error{Kind: KindNotImplemented}
)

Sentinel values for errors.Is matching by kind.

Functions

func Load 

func Load(path string) (*gruby.Table, error)

Load reads a Parquet file at path into a go-ruby-arrow table, mirroring red-arrow's Arrow::Table.load format dispatch for the ".parquet" extension.

func ReadTableBytes 

func ReadTableBytes(data []byte) (*gruby.Table, error)

ReadTableBytes reads a whole Parquet file held in memory into a go-ruby-arrow table, a convenience over NewArrowFileReader + ArrowFileReader.ReadTable.

func Save 

func Save(t *gruby.Table, path string, opts ...WriteOption) error

Save writes a go-ruby-arrow table to a Parquet file at path, mirroring red-arrow's Arrow::Table#save format dispatch for the ".parquet" extension.

func WriteTable 

func WriteTable(t *gruby.Table, path string, opts ...WriteOption) error

WriteTable writes a go-ruby-arrow table to a Parquet file at path (Parquet::Writer.write(table, path)).

func WriteTableTo 

func WriteTableTo(w io.Writer, t *gruby.Table, opts ...WriteOption) error

WriteTableTo writes a single go-ruby-arrow table to w as a complete Parquet file (open, write, close), mirroring Parquet::Writer.write to an IO.

Types

type ArrowFileReader 

type ArrowFileReader struct {
	// contains filtered or unexported fields
}

ArrowFileReader is the pure-Go counterpart of Parquet::ArrowFileReader — a random-access Parquet reader that yields go-ruby-arrow tables. Construct one from an in-memory reader with NewArrowFileReader or from a path with OpenArrowFileReader, read with ArrowFileReader.ReadTable / ArrowFileReader.ReadRowGroup, and ArrowFileReader.Close it when done.

func NewArrowFileReader 

func NewArrowFileReader(r parquet.ReaderAtSeeker) (*ArrowFileReader, error)

NewArrowFileReader opens a Parquet reader over an in-memory random-access source (Parquet::ArrowFileReader.new with an IO). A *bytes.Reader or *os.File satisfies parquet.ReaderAtSeeker.

func OpenArrowFileReader 

func OpenArrowFileReader(path string) (*ArrowFileReader, error)

OpenArrowFileReader opens the Parquet file at path (Parquet::ArrowFileReader.new with a path). The whole file is buffered into memory and the OS file handle is released before returning, so no handle outlives the call — on Windows the file can then always be deleted, even while the reader is still in use.

func (*ArrowFileReader) Close 

func (r *ArrowFileReader) Close() error

Close releases the reader (Parquet::ArrowFileReader#close). The source is always in memory, so there is no OS file handle to release. It is idempotent.

func (*ArrowFileReader) NumRowGroups 

func (r *ArrowFileReader) NumRowGroups() int

NumRowGroups returns the number of row groups (Parquet::ArrowFileReader#n_row_groups).

func (*ArrowFileReader) NumRows 

func (r *ArrowFileReader) NumRows() int64

NumRows returns the total number of rows across all row groups (Parquet::ArrowFileReader#n_rows).

func (*ArrowFileReader) ReadRowGroup 

func (r *ArrowFileReader) ReadRowGroup(idx int) (*gruby.Table, error)

ReadRowGroup reads a single row group by index into a go-ruby-arrow table (Parquet::ArrowFileReader#read_row_group). An out-of-range index yields an *Error of KindIndex.

func (*ArrowFileReader) ReadTable 

func (r *ArrowFileReader) ReadTable() (*gruby.Table, error)

ReadTable reads every row group into one go-ruby-arrow table (Parquet::ArrowFileReader#read_table).

func (*ArrowFileReader) Schema 

func (r *ArrowFileReader) Schema() (*gruby.Schema, error)

Schema returns the file's schema as a go-ruby-arrow schema (Parquet::ArrowFileReader#schema).

type ArrowFileWriter 

type ArrowFileWriter struct {
	// contains filtered or unexported fields
}

ArrowFileWriter is the pure-Go counterpart of Parquet::ArrowFileWriter — a streaming Parquet writer bound to an io.Writer and an Arrow schema. Write one or more go-ruby-arrow tables, then ArrowFileWriter.Close to flush the footer.

func NewArrowFileWriter 

func NewArrowFileWriter(w io.Writer, schema *gruby.Schema, opts ...WriteOption) (*ArrowFileWriter, error)

NewArrowFileWriter opens a Parquet writer over w for tables matching schema (Parquet::ArrowFileWriter.new). The write options select compression, row-group size and dictionary encoding.

func (*ArrowFileWriter) Close 

func (w *ArrowFileWriter) Close() error

Close flushes the Parquet footer and releases the writer (Parquet::ArrowFileWriter#close). It is idempotent.

func (*ArrowFileWriter) Write 

func (w *ArrowFileWriter) Write(t *gruby.Table) error

Write appends a go-ruby-arrow table to the file, chunked into row groups of the configured size (Parquet::ArrowFileWriter#write_table). The table's schema must match the writer's schema.

type Compression 

type Compression int

Compression selects the Parquet column-chunk compression codec, mirroring red-parquet's per-file compression symbols (:uncompressed, :snappy, :gzip, :zstd) passed to Parquet::WriterProperties#set_compression. 

const (
	// Uncompressed stores column chunks with no compression.
	Uncompressed Compression = iota
	// Snappy is red-parquet's default codec — fast, moderate ratio.
	Snappy
	// Gzip (DEFLATE) trades speed for a better ratio.
	Gzip
	// Zstd offers a strong ratio at competitive speed.
	Zstd
)

func ParseCompression 

func ParseCompression(name string) (Compression, error)

ParseCompression resolves a red-parquet compression symbol/string (case- and leading-colon-insensitive, e.g. "snappy", ":gzip", "ZSTD") to a Compression. An unrecognized name yields an *Error of KindArgument.

func (Compression) String 

func (c Compression) String() string

String returns the Ruby symbol name of the codec (without the leading colon), e.g. "snappy", matching red-parquet's compression names.

type Error 

type Error struct {
	Kind ErrorKind
	Msg  string
	Err  error
}

Error is the pure-Go counterpart of red-parquet's Parquet::Error exception tree. It carries the ErrorKind (so the exact Ruby class can be reconstructed) and an optional wrapped cause, and it participates in errors.Is/As via Error.Is and Error.Unwrap.

func (*Error) Error 

func (e *Error) Error() string

Error implements the error interface.

func (*Error) Is 

func (e *Error) Is(target error) bool

Is reports whether target is an *Error of the same ErrorKind, letting callers write errors.Is(err, parquet.ErrIO) against the sentinel values.

func (*Error) RubyClass 

func (e *Error) RubyClass() string

RubyClass returns the fully-qualified Ruby exception class name a faithful host raises for this error, mirroring what red-parquet raises.

func (*Error) Unwrap 

func (e *Error) Unwrap() error

Unwrap returns the wrapped cause, if any, so errors.Is/As traverse it.

type ErrorKind 

type ErrorKind int

ErrorKind identifies which node of red-parquet's exception tree an Error corresponds to. red-parquet raises Ruby exception classes (largely inherited from Arrow's); the kind records which one so a host (rbgo) can re-raise the faithful class. 

const (
	// KindError is the base Parquet::Error (a StandardError in Ruby).
	KindError ErrorKind = iota
	// KindType maps to Ruby's TypeError — a value did not fit the column type.
	KindType
	// KindIndex maps to Ruby's IndexError — an out-of-range row group / column.
	KindIndex
	// KindArgument maps to Ruby's ArgumentError — a malformed call or option.
	KindArgument
	// KindIO maps to Parquet::Error::Io — a Parquet read/write failure.
	KindIO
	// KindNotImplemented maps to Ruby's NotImplementedError.
	KindNotImplemented
)

type WriteOption 

type WriteOption func(*writeConfig)

WriteOption configures the Parquet writer, mirroring the knobs red-parquet exposes through Parquet::WriterProperties (compression, row-group size and dictionary encoding).

func WithCompression 

func WithCompression(c Compression) WriteOption

WithCompression sets the column-chunk compression codec (Parquet::WriterProperties#set_compression).

func WithDictionary 

func WithDictionary(enabled bool) WriteOption

WithDictionary enables or disables dictionary encoding (Parquet::WriterProperties#set_enable_dictionary).

func WithRowGroupSize 

func WithRowGroupSize(n int64) WriteOption

WithRowGroupSize sets the maximum number of rows per row group (Parquet::ArrowFileWriter#write_table chunk_size). A non-positive size falls back to DefaultRowGroupSize.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.