paimon

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 27, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package paimon provides a Go binding for Apache Paimon Rust.

This binding uses purego and libffi to call into the paimon-c shared library. The pre-built shared library is embedded in the package and automatically loaded at runtime — no manual build step needed.

This package requires CGO because it imports the arrow-go cdata package for Arrow C Data Interface support.

Basic usage: 

// Create a catalog with options
catalog, err := paimon.NewCatalog(map[string]string{
	"warehouse": "/path/to/warehouse",
})
if err != nil { log.Fatal(err) }
defer catalog.Close()

table, err := catalog.GetTable(paimon.NewIdentifier("default", "my_table"))
if err != nil { log.Fatal(err) }
defer table.Close()

For S3 or OSS warehouses, pass the appropriate credentials: 

// S3
catalog, _ := paimon.NewCatalog(map[string]string{
	"warehouse":            "s3://bucket/warehouse",
	"s3.access-key-id":     "...",
	"s3.secret-access-key": "...",
	"s3.region":            "us-east-1",
})

// OSS
catalog, _ := paimon.NewCatalog(map[string]string{
	"warehouse":            "oss://bucket/warehouse",
	"fs.oss.accessKeyId":     "...",
	"fs.oss.accessKeySecret": "...",
	"fs.oss.endpoint":        "oss-cn-hangzhou.aliyuncs.com",
})

Index

Constants

This section is empty.

Variables

View Source 
var ErrClosed = errors.New("paimon: use of closed resource")

ErrClosed is returned when an operation is attempted on a closed resource.

Functions

This section is empty.

Types

type Bytes 

type Bytes []byte

Bytes represents a binary value. Usage: table.PredicateEqual("data", paimon.Bytes(someSlice))

type Catalog 

type Catalog struct {
	// contains filtered or unexported fields
}

Catalog wraps a paimon Catalog.

func NewCatalog 

func NewCatalog(options map[string]string) (*Catalog, error)

NewCatalog creates a new Catalog using the CatalogFactory with the given options. The catalog type is determined by the "metastore" option (default: "filesystem").

Common options:

  • "warehouse": The warehouse path (required)
  • "metastore": Catalog type - "filesystem" (default) or "rest"
  • "uri": REST catalog server URI (required for REST catalog)
  • "s3.access-key-id", "s3.secret-access-key", "s3.region": S3 credentials
  • "fs.oss.accessKeyId", "fs.oss.accessKeySecret", "fs.oss.endpoint": OSS credentials

func (*Catalog) Close 

func (c *Catalog) Close()

Close releases the catalog resources. Safe to call multiple times.

func (*Catalog) GetTable 

func (c *Catalog) GetTable(id Identifier) (*Table, error)

GetTable retrieves a table from the catalog using the given identifier.

type DataSplit 

type DataSplit struct {
	// contains filtered or unexported fields
}

DataSplit identifies a single data split within a plan. DataSplits keep the underlying plan data alive via GC-attached reference counting, so they are safe to use independently.

type Date 

type Date int32

Date represents a date value as epoch days since 1970-01-01. Usage: table.PredicateEqual("dt", paimon.Date(19000))

type Datum 

type Datum struct {
	// contains filtered or unexported fields
}

Datum is a typed literal value for predicate comparison. The internal representation is hidden to allow future changes (e.g. switching to opaque handles) without breaking callers.

func BoolDatum 

func BoolDatum(v bool) Datum

BoolDatum creates a boolean datum.

func DoubleDatum 

func DoubleDatum(v float64) Datum

DoubleDatum creates a double datum.

func FloatDatum 

func FloatDatum(v float32) Datum

FloatDatum creates a float datum.

func IntDatum 

func IntDatum(v int32) Datum

IntDatum creates an int datum.

func LongDatum 

func LongDatum(v int64) Datum

LongDatum creates a long (bigint) datum.

func SmallIntDatum 

func SmallIntDatum(v int16) Datum

SmallIntDatum creates a smallint datum.

func StringDatum 

func StringDatum(v string) Datum

StringDatum creates a string datum.

func TinyIntDatum 

func TinyIntDatum(v int8) Datum

TinyIntDatum creates a tinyint datum.

type Decimal 

type Decimal struct {
	Lo        int64 // low 64 bits of unscaled i128 (unsigned interpretation)
	Hi        int64 // high 64 bits of unscaled i128 (sign extension)
	Precision uint32
	Scale     uint32
}

Decimal represents a fixed-precision decimal value up to DECIMAL(38, s).

The unscaled value is stored as a little-endian i128 split into two int64 halves: Lo (low 64 bits, unsigned interpretation) and Hi (high 64 bits, sign-extended). For values that fit in int64, use NewDecimal.

Usage: 

paimon.NewDecimal(12345, 10, 2)          // 123.45 as DECIMAL(10,2)
paimon.Decimal{Lo: lo, Hi: hi, ...}      // full i128

func NewDecimal 

func NewDecimal(unscaled int64, precision, scale uint32) Decimal

NewDecimal creates a Decimal from an int64 unscaled value. For unscaled values that exceed int64 range, construct Decimal directly with Lo/Hi fields.

type Error 

type Error struct {
	// contains filtered or unexported fields
}

Error represents a paimon error with code and message.

func (*Error) Code 

func (e *Error) Code() ErrorCode

Code returns the error code.

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Message 

func (e *Error) Message() string

Message returns the error message.

type ErrorCode 

type ErrorCode int32

ErrorCode represents categories of errors from paimon. 

const (
	CodeUnexpected   ErrorCode = 0
	CodeUnsupported  ErrorCode = 1
	CodeNotFound     ErrorCode = 2
	CodeAlreadyExist ErrorCode = 3
	CodeInvalidInput ErrorCode = 4
	CodeIoError      ErrorCode = 5
)

type FFI 

type FFI[T any] struct {
	// contains filtered or unexported fields
}

FFI is a generic type-safe wrapper for a foreign function.

type Identifier 

type Identifier struct {
	// contains filtered or unexported fields
}

Identifier identifies a table by database and object name.

func NewIdentifier 

func NewIdentifier(database, object string) Identifier

NewIdentifier creates a new Identifier with the given database and object name.

type LocalZonedTimestamp 

type LocalZonedTimestamp struct {
	Millis int64
	Nanos  int32
}

LocalZonedTimestamp represents a timestamp with local timezone semantics. Usage: table.PredicateEqual("lzts", paimon.LocalZonedTimestamp{Millis: 1700000000000, Nanos: 0})

type Plan 

type Plan struct {
	// contains filtered or unexported fields
}

Plan holds the result of a table scan, containing data splits to read.

func (*Plan) Close 

func (p *Plan) Close()

Close releases the plan resources. Safe to call multiple times. DataSplits obtained from Splits() remain valid after Close.

func (*Plan) NumSplits 

func (p *Plan) NumSplits() int

NumSplits returns the number of data splits in this plan.

func (*Plan) Splits 

func (p *Plan) Splits() []DataSplit

Splits returns all data splits in this plan. The returned DataSplits keep the underlying plan data alive via GC-attached reference counting, so they remain valid even after Plan.Close() is called.

type Predicate 

type Predicate struct {
	// contains filtered or unexported fields
}

Predicate is an opaque filter predicate for scan planning.

func (*Predicate) And 

func (p *Predicate) And(other *Predicate) (*Predicate, error)

And combines this predicate with another using AND. Consumes both predicates (callers must NOT close either after this call).

func (*Predicate) Close 

func (p *Predicate) Close()

Close releases the predicate resources. Safe to call multiple times. Note: predicates passed to WithFilter or combinators (And/Or/Not) are consumed and should NOT be closed by the caller.

func (*Predicate) Not 

func (p *Predicate) Not() (*Predicate, error)

Not negates this predicate. Consumes the input (caller must NOT close it after this call).

func (*Predicate) Or 

func (p *Predicate) Or(other *Predicate) (*Predicate, error)

Or combines this predicate with another using OR. Consumes both predicates (callers must NOT close either after this call).

type PredicateBuilder 

type PredicateBuilder struct {
	// contains filtered or unexported fields
}

PredicateBuilder creates filter predicates for a table. It holds a Go-level reference to the Table and does not own any C resources, so there is no Close() method.

func (*PredicateBuilder) Eq 

func (pb *PredicateBuilder) Eq(column string, value any) (*Predicate, error)

Eq creates an equality predicate: column = value.

func (*PredicateBuilder) Ge 

func (pb *PredicateBuilder) Ge(column string, value any) (*Predicate, error)

Ge creates a greater-or-equal predicate: column >= value.

func (*PredicateBuilder) Gt 

func (pb *PredicateBuilder) Gt(column string, value any) (*Predicate, error)

Gt creates a greater-than predicate: column > value.

func (*PredicateBuilder) In 

func (pb *PredicateBuilder) In(column string, values ...any) (*Predicate, error)

In creates an IN predicate: column IN (values...).

func (*PredicateBuilder) IsNotNull 

func (pb *PredicateBuilder) IsNotNull(column string) (*Predicate, error)

IsNotNull creates an IS NOT NULL predicate.

func (*PredicateBuilder) IsNull 

func (pb *PredicateBuilder) IsNull(column string) (*Predicate, error)

IsNull creates an IS NULL predicate.

func (*PredicateBuilder) Le 

func (pb *PredicateBuilder) Le(column string, value any) (*Predicate, error)

Le creates a less-or-equal predicate: column <= value.

func (*PredicateBuilder) Lt 

func (pb *PredicateBuilder) Lt(column string, value any) (*Predicate, error)

Lt creates a less-than predicate: column < value.

func (*PredicateBuilder) NotEq 

func (pb *PredicateBuilder) NotEq(column string, value any) (*Predicate, error)

NotEq creates a not-equal predicate: column != value.

func (*PredicateBuilder) NotIn 

func (pb *PredicateBuilder) NotIn(column string, values ...any) (*Predicate, error)

NotIn creates a NOT IN predicate: column NOT IN (values...).

type ReadBuilder 

type ReadBuilder struct {
	// contains filtered or unexported fields
}

ReadBuilder creates TableScan and TableRead instances.

func (*ReadBuilder) Close 

func (rb *ReadBuilder) Close()

Close releases the read builder resources. Safe to call multiple times.

func (*ReadBuilder) NewRead 

func (rb *ReadBuilder) NewRead() (*TableRead, error)

NewRead creates a TableRead for reading data from splits.

func (*ReadBuilder) NewScan 

func (rb *ReadBuilder) NewScan() (*TableScan, error)

NewScan creates a TableScan for planning which data files to read.

func (*ReadBuilder) WithFilter 

func (rb *ReadBuilder) WithFilter(p *Predicate) error

WithFilter sets a filter predicate for scan planning and read-side pruning.

The predicate is used in two phases:

  • Scan planning: prunes partitions, buckets, and data files based on file-level statistics (min/max). This is conservative — files whose statistics are inconclusive are kept.
  • Read-side: applies row-level filtering via Parquet native row filters for supported leaf predicates (Eq, NotEq, Lt, Le, Gt, Ge, IsNull, IsNotNull, In, NotIn).

Row-level filtering is exact for most common types (Bool, Int, Long, Float, Double, String, Date, Decimal, Binary). However, the following cases are NOT filtered at the row level and may return non-matching rows:

  • Compound predicates (And/Or/Not) — not yet implemented for row-level filtering.
  • Time, Timestamp, and LocalZonedTimestamp columns (not yet implemented).
  • Schema-evolution: the predicate column does not exist in older data files.
  • Data-evolution mode (data-evolution.enabled = true).

In these cases callers should apply residual filtering on the returned records.

The predicate is consumed (ownership transferred to the read builder); the caller must NOT close it after this call. Passing nil is a no-op.

func (*ReadBuilder) WithProjection 

func (rb *ReadBuilder) WithProjection(columns []string) error

WithProjection sets column projection by name. Output order follows the caller-specified order. Unknown or duplicate names cause NewRead() to fail; an empty list is a valid zero-column projection.

type RecordBatchReader 

type RecordBatchReader struct {
	// contains filtered or unexported fields
}

RecordBatchReader iterates over Arrow record batches one at a time via the Arrow C Data Interface (zero-copy). Call NextRecord to advance and Close when done.

func (*RecordBatchReader) Close 

func (r *RecordBatchReader) Close()

Close releases the underlying C record batch readers. Safe to call multiple times.

func (*RecordBatchReader) NextRecord 

func (r *RecordBatchReader) NextRecord() (arrow.Record, error)

NextRecord returns the next Arrow record, or io.EOF when iteration is complete. The underlying C batch is imported via the Arrow C Data Interface and released automatically — the caller only needs to call Release on the returned arrow.Record when done.

type Table 

type Table struct {
	// contains filtered or unexported fields
}

Table represents a paimon table.

func (*Table) Close 

func (t *Table) Close()

Close releases the table resources. Safe to call multiple times.

func (*Table) NewReadBuilder 

func (t *Table) NewReadBuilder() (*ReadBuilder, error)

NewReadBuilder creates a ReadBuilder for this table.

func (*Table) PredicateBuilder 

func (t *Table) PredicateBuilder() *PredicateBuilder

PredicateBuilder returns a builder for creating filter predicates on this table.

type TableRead 

type TableRead struct {
	// contains filtered or unexported fields
}

TableRead reads data from splits produced by a TableScan.

func (*TableRead) Close 

func (tr *TableRead) Close()

Close releases the table read resources. Safe to call multiple times.

func (*TableRead) NewRecordBatchReader 

func (tr *TableRead) NewRecordBatchReader(splits []DataSplit) (*RecordBatchReader, error)

NewRecordBatchReader creates a RecordBatchReader that iterates over Arrow record batches for the given data splits. The splits can be non-contiguous and in any order. All splits must originate from the same Plan.

type TableScan 

type TableScan struct {
	// contains filtered or unexported fields
}

TableScan scans a table and produces a Plan containing data splits.

func (*TableScan) Close 

func (ts *TableScan) Close()

Close releases the table scan resources. Safe to call multiple times.

func (*TableScan) Plan 

func (ts *TableScan) Plan() (*Plan, error)

Plan executes the scan and returns a Plan containing data splits to read.

type Time 

type Time int32

Time represents a time-of-day value as milliseconds since midnight. Usage: table.PredicateEqual("t", paimon.Time(3600000))

type Timestamp 

type Timestamp struct {
	Millis int64
	Nanos  int32
}

Timestamp represents a timestamp without timezone (millis + sub-millis nanos). Usage: table.PredicateEqual("ts", paimon.Timestamp{Millis: 1700000000000, Nanos: 0})

Source Files

View all Source files

Directories

Path Synopsis
Package predicate provides convenience functions for combining Paimon filter predicates.
 Package predicate provides convenience functions for combining Paimon filter predicates.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.