iceberg

package module
v0.0.0-...-4929eea Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 9, 2024 License: Apache-2.0 Imports: 29 Imported by: 1

README

Iceberg Golang

Go Reference

iceberg is a Golang implementation of the Iceberg table spec.

Build From Source

Prerequisites
  • Go 1.21 or later
Build
$ git clone https://github.com/apache/iceberg-go.git
$ cd iceberg-go/cmd/iceberg && go build .

Feature Support / Roadmap

FileSystem Support
Filesystem Type Supported
S3 X
Google Cloud Storage
Azure Blob Storage
Local Filesystem X
Metadata
Operation Supported
Get Schema X
Get Snapshots X
Get Sort Orders X
Get Partition Specs X
Get Manifests X
Create New Manifests X
Plan Scan
Plan Scan for Snapshot
Catalog Support
Operation REST Hive DynamoDB Glue
Load Table X
List Tables X
Create Table
Update Current Snapshot
Create New Snapshot
Rename Table
Drop Table
Alter Table
Set Table Properties
Create Namespace
Drop Namespace
Set Namespace Properties
Read/Write Data Support
  • No intrinsic support for reading/writing data yet
  • Data can be manually read currently by retrieving data files via Manifests.
  • Plan to add Apache Arrow support eventually.

Get in Touch

Documentation

Index

Constants

View Source
const (
	InitialPartitionSpecID = 0
)

Variables

View Source
var (
	ErrInvalidTypeString       = errors.New("invalid type")
	ErrNotImplemented          = errors.New("not implemented")
	ErrInvalidArgument         = errors.New("invalid argument")
	ErrInvalidSchema           = errors.New("invalid schema")
	ErrInvalidTransform        = errors.New("invalid transform syntax")
	ErrType                    = errors.New("type error")
	ErrBadCast                 = errors.New("could not cast value")
	ErrBadLiteral              = errors.New("invalid literal value")
	ErrInvalidBinSerialization = errors.New("invalid binary serialization")
)
View Source
var PositionalDeleteSchema = NewSchema(0,
	NestedField{ID: 2147483546, Type: PrimitiveTypes.String, Name: "file_path", Required: true},
	NestedField{ID: 2147483545, Type: PrimitiveTypes.Int32, Name: "pos", Required: true},
)
View Source
var PrimitiveTypes = struct {
	Bool        PrimitiveType
	Int32       PrimitiveType
	Int64       PrimitiveType
	Float32     PrimitiveType
	Float64     PrimitiveType
	Date        PrimitiveType
	Time        PrimitiveType
	Timestamp   PrimitiveType
	TimestampTz PrimitiveType
	String      PrimitiveType
	Binary      PrimitiveType
	UUID        PrimitiveType
}{
	Bool:        BooleanType{},
	Int32:       Int32Type{},
	Int64:       Int64Type{},
	Float32:     Float32Type{},
	Float64:     Float64Type{},
	Date:        DateType{},
	Time:        TimeType{},
	Timestamp:   TimestampType{},
	TimestampTz: TimestampTzType{},
	String:      StringType{},
	Binary:      BinaryType{},
	UUID:        UUIDType{},
}
View Source
var UnpartitionedSpec = &PartitionSpec{id: 0}

UnpartitionedSpec is the default unpartitioned spec which can be used for comparisons or to just provide a convenience for referencing the same unpartitioned spec object.

Functions

func ExpressionEvaluator

func ExpressionEvaluator(s *Schema, unbound BooleanExpression, caseSensitive bool) (func(structLike) (bool, error), error)

ExpressionEvaluator returns a function which can be used to evaluate a given expression as long as a structlike value is passed which operates like and matches the passed in schema.

func IndexByID

func IndexByID(schema *Schema) (map[int]NestedField, error)

IndexByID performs a post-order traversal of the given schema and returns a mapping from field ID to field.

func IndexByName

func IndexByName(schema *Schema) (map[string]int, error)

IndexByName performs a post-order traversal of the schema and returns a mapping from field name to field ID.

func IndexNameByID

func IndexNameByID(schema *Schema) (map[int]string, error)

IndexNameByID performs a post-order traversal of the schema and returns a mapping from field ID to field name.

func IndexParents

func IndexParents(schema *Schema) (map[int]int, error)

IndexParents generates an index of field IDs to their parent field IDs. Root fields are not indexed

func Version

func Version() string

func Visit

func Visit[T any](sc *Schema, visitor SchemaVisitor[T]) (res T, err error)

Visit accepts a visitor and performs a post-order traversal of the given schema.

func VisitBoundPredicate

func VisitBoundPredicate[T any](e BoundPredicate, visitor BoundBooleanExprVisitor[T]) T

VisitBoundPredicate uses a BoundBooleanExprVisitor to call the appropriate method based on the type of operation in the predicate. This is a convenience function for implementing the VisitBound method of a BoundBooleanExprVisitor by simply calling iceberg.VisitBoundPredicate(pred, this).

func VisitExpr

func VisitExpr[T any](expr BooleanExpression, visitor BooleanExprVisitor[T]) (res T, err error)

VisitExpr is a convenience function to use a given visitor to visit all parts of a boolean expression in-order. Values returned from the methods are passed to the subsequent methods, effectively "bubbling up" the results.

func VisitSchemaWithPartner

func VisitSchemaWithPartner[T, P any](sc *Schema, partner P, visitor SchemaWithPartnerVisitor[T, P], accessor PartnerAccessor[P]) (res T, err error)

Types

type AboveMaxLiteral

type AboveMaxLiteral interface {
	Literal
	// contains filtered or unexported methods
}

AboveMaxLiteral represents values that are above the maximum for their type such as values > math.MaxInt32 for an Int32Literal

type AfterFieldVisitor

type AfterFieldVisitor interface {
	AfterField(field NestedField)
}

type AfterListElementVisitor

type AfterListElementVisitor interface {
	AfterListElement(elem NestedField)
}

type AfterMapKeyVisitor

type AfterMapKeyVisitor interface {
	AfterMapKey(key NestedField)
}

type AfterMapValueVisitor

type AfterMapValueVisitor interface {
	AfterMapValue(value NestedField)
}

type AlwaysFalse

type AlwaysFalse struct{}

AlwaysFalse is the boolean expression "False"

func (AlwaysFalse) Equals

func (AlwaysFalse) Equals(other BooleanExpression) bool

func (AlwaysFalse) Negate

func (AlwaysFalse) Negate() BooleanExpression

func (AlwaysFalse) Op

func (AlwaysFalse) Op() Operation

func (AlwaysFalse) String

func (AlwaysFalse) String() string

type AlwaysTrue

type AlwaysTrue struct{}

AlwaysTrue is the boolean expression "True"

func (AlwaysTrue) Equals

func (AlwaysTrue) Equals(other BooleanExpression) bool

func (AlwaysTrue) Negate

func (AlwaysTrue) Negate() BooleanExpression

func (AlwaysTrue) Op

func (AlwaysTrue) Op() Operation

func (AlwaysTrue) String

func (AlwaysTrue) String() string

type AndExpr

type AndExpr struct {
	// contains filtered or unexported fields
}

func (AndExpr) Equals

func (a AndExpr) Equals(other BooleanExpression) bool

func (AndExpr) Negate

func (a AndExpr) Negate() BooleanExpression

func (AndExpr) Op

func (AndExpr) Op() Operation

func (AndExpr) String

func (a AndExpr) String() string

type BeforeFieldVisitor

type BeforeFieldVisitor interface {
	BeforeField(field NestedField)
}

type BeforeListElementVisitor

type BeforeListElementVisitor interface {
	BeforeListElement(elem NestedField)
}

type BeforeMapKeyVisitor

type BeforeMapKeyVisitor interface {
	BeforeMapKey(key NestedField)
}

type BeforeMapValueVisitor

type BeforeMapValueVisitor interface {
	BeforeMapValue(value NestedField)
}

type BelowMinLiteral

type BelowMinLiteral interface {
	Literal
	// contains filtered or unexported methods
}

BelowMinLiteral represents values that are below the minimum for their type such as values < math.MinInt32 for an Int32Literal

type BinaryLiteral

type BinaryLiteral []byte

func (BinaryLiteral) Comparator

func (BinaryLiteral) Comparator() Comparator[[]byte]

func (BinaryLiteral) Equals

func (b BinaryLiteral) Equals(other Literal) bool

func (BinaryLiteral) MarshalBinary

func (b BinaryLiteral) MarshalBinary() (data []byte, err error)

func (BinaryLiteral) String

func (b BinaryLiteral) String() string

func (BinaryLiteral) To

func (b BinaryLiteral) To(typ Type) (Literal, error)

func (BinaryLiteral) Type

func (b BinaryLiteral) Type() Type

func (*BinaryLiteral) UnmarshalBinary

func (b *BinaryLiteral) UnmarshalBinary(data []byte) error

func (BinaryLiteral) Value

func (b BinaryLiteral) Value() []byte

type BinaryType

type BinaryType struct{}

func (BinaryType) Equals

func (BinaryType) Equals(other Type) bool

func (BinaryType) String

func (BinaryType) String() string

func (BinaryType) Type

func (BinaryType) Type() string

type BoolLiteral

type BoolLiteral bool

func (BoolLiteral) Comparator

func (BoolLiteral) Comparator() Comparator[bool]

func (BoolLiteral) Equals

func (b BoolLiteral) Equals(l Literal) bool

func (BoolLiteral) MarshalBinary

func (b BoolLiteral) MarshalBinary() (data []byte, err error)

func (BoolLiteral) String

func (b BoolLiteral) String() string

func (BoolLiteral) To

func (b BoolLiteral) To(t Type) (Literal, error)

func (BoolLiteral) Type

func (b BoolLiteral) Type() Type

func (*BoolLiteral) UnmarshalBinary

func (b *BoolLiteral) UnmarshalBinary(data []byte) error

func (BoolLiteral) Value

func (b BoolLiteral) Value() bool

type BooleanExprVisitor

type BooleanExprVisitor[T any] interface {
	VisitTrue() T
	VisitFalse() T
	VisitNot(childResult T) T
	VisitAnd(left, right T) T
	VisitOr(left, right T) T
	VisitUnbound(UnboundPredicate) T
	VisitBound(BoundPredicate) T
}

BooleanExprVisitor is an interface for recursively visiting the nodes of a boolean expression

type BooleanExpression

type BooleanExpression interface {
	fmt.Stringer
	Op() Operation
	Negate() BooleanExpression
	Equals(BooleanExpression) bool
}

BooleanExpression represents a full expression which will evaluate to a boolean value such as GreaterThan or StartsWith, etc.

func BindExpr

func BindExpr(s *Schema, expr BooleanExpression, caseSensitive bool) (BooleanExpression, error)

BindExpr recursively binds each portion of an expression using the provided schema. Because the expression can end up being simplified to just AlwaysTrue/AlwaysFalse, this returns a BooleanExpression.

func IsIn

func IsIn[T LiteralType](t UnboundTerm, vals ...T) BooleanExpression

IsIn is a convenience wrapper for constructing an unbound set predicate for OpIn. It returns a BooleanExpression instead of an UnboundPredicate because depending on the arguments, it can automatically reduce to AlwaysFalse or AlwaysTrue (if given no values for examples). It may also reduce to EqualTo if only one value is provided.

Will panic if t is nil

func NewAnd

func NewAnd(left, right BooleanExpression, addl ...BooleanExpression) BooleanExpression

NewAnd will construct a new AndExpr, allowing the caller to provide potentially more than just two arguments which will be folded to create an appropriate expression tree. i.e. NewAnd(a, b, c, d) becomes AndExpr(a, AndExpr(b, AndExpr(c, d)))

Slight optimizations are performed on creation if either argument is AlwaysFalse or AlwaysTrue by performing reductions. If any argument is AlwaysFalse, then everything will get folded to a return of AlwaysFalse. If an argument is AlwaysTrue, then the other argument will be returned directly rather than creating an AndExpr.

Will panic if any argument is nil

func NewNot

NewNot creates a BooleanExpression representing a "Not" operation on the given argument. It will optimize slightly though:

If the argument is AlwaysTrue or AlwaysFalse, the appropriate inverse expression will be returned directly. If the argument is itself a NotExpr, then the child will be returned rather than NotExpr(NotExpr(child)).

func NewOr

func NewOr(left, right BooleanExpression, addl ...BooleanExpression) BooleanExpression

NewOr will construct a new OrExpr, allowing the caller to provide potentially more than just two arguments which will be folded to create an appropriate expression tree. i.e. NewOr(a, b, c, d) becomes OrExpr(a, OrExpr(b, OrExpr(c, d)))

Slight optimizations are performed on creation if either argument is AlwaysFalse or AlwaysTrue by performing reductions. If any argument is AlwaysTrue, then everything will get folded to a return of AlwaysTrue. If an argument is AlwaysFalse, then the other argument will be returned directly rather than creating an OrExpr.

Will panic if any argument is nil

func NotIn

func NotIn[T LiteralType](t UnboundTerm, vals ...T) BooleanExpression

NotIn is a convenience wrapper for constructing an unbound set predicate for OpNotIn. It returns a BooleanExpression instead of an UnboundPredicate because depending on the arguments, it can automatically reduce to AlwaysFalse or AlwaysTrue (if given no values for examples). It may also reduce to NotEqualTo if only one value is provided.

Will panic if t is nil

func RewriteNotExpr

func RewriteNotExpr(expr BooleanExpression) (BooleanExpression, error)

RewriteNotExpr rewrites a boolean expression to remove "Not" nodes from the expression tree. This is because Projections assume there are no "not" nodes.

Not nodes will be replaced with simply calling `Negate` on the child in the tree.

func SetPredicate

func SetPredicate(op Operation, t UnboundTerm, lits []Literal) BooleanExpression

SetPredicate creates a boolean expression representing a predicate that uses a set of literals as the argument, like In or NotIn. Duplicate literals will be folded into a set, only maintaining the unique literals.

Will panic if op is not a valid Set operation

type BooleanType

type BooleanType struct{}

func (BooleanType) Equals

func (BooleanType) Equals(other Type) bool

func (BooleanType) String

func (BooleanType) String() string

func (BooleanType) Type

func (BooleanType) Type() string

type BoundBooleanExprVisitor

type BoundBooleanExprVisitor[T any] interface {
	BooleanExprVisitor[T]

	VisitIn(BoundTerm, Set[Literal]) T
	VisitNotIn(BoundTerm, Set[Literal]) T
	VisitIsNan(BoundTerm) T
	VisitNotNan(BoundTerm) T
	VisitIsNull(BoundTerm) T
	VisitNotNull(BoundTerm) T
	VisitEqual(BoundTerm, Literal) T
	VisitNotEqual(BoundTerm, Literal) T
	VisitGreaterEqual(BoundTerm, Literal) T
	VisitGreater(BoundTerm, Literal) T
	VisitLessEqual(BoundTerm, Literal) T
	VisitLess(BoundTerm, Literal) T
	VisitStartsWith(BoundTerm, Literal) T
	VisitNotStartsWith(BoundTerm, Literal) T
}

BoundBooleanExprVisitor builds on BooleanExprVisitor by adding interface methods for visiting bound expressions, because we do casting of literals during binding you can assume that the BoundTerm and the Literal passed to a method have the same type.

type BoundLiteralPredicate

type BoundLiteralPredicate interface {
	BoundPredicate

	Literal() Literal
	AsUnbound(Reference, Literal) UnboundPredicate
}

BoundLiteralPredicate represents a bound boolean expression that utilizes a single literal as an argument, such as Equals or StartsWith.

type BoundPredicate

type BoundPredicate interface {
	BooleanExpression
	Ref() BoundReference
	Term() BoundTerm
}

BoundPredicate is a boolean predicate expression which has been bound to a schema. The underlying reference and term can be retrieved from it.

type BoundReference

type BoundReference interface {
	BoundTerm

	Field() NestedField
	Pos() int
}

BoundReference is a named reference that has been bound to a particular field in a given schema.

type BoundSetPredicate

type BoundSetPredicate interface {
	BoundPredicate

	Literals() Set[Literal]
	AsUnbound(Reference, []Literal) UnboundPredicate
}

BoundSetPredicate is a bound expression that utilizes a set of literals such as In or NotIn

type BoundTerm

type BoundTerm interface {
	Term

	Equals(BoundTerm) bool
	Ref() BoundReference
	Type() Type
	// contains filtered or unexported methods
}

BoundTerm is a simple expression (typically a reference) that evaluates to a value and has been bound to a schema.

type BoundTransform

type BoundTransform struct {
	// contains filtered or unexported fields
}

func (*BoundTransform) Equals

func (b *BoundTransform) Equals(other BoundTerm) bool

func (*BoundTransform) Ref

func (b *BoundTransform) Ref() BoundReference

func (*BoundTransform) String

func (b *BoundTransform) String() string

func (*BoundTransform) Type

func (b *BoundTransform) Type() Type

type BoundUnaryPredicate

type BoundUnaryPredicate interface {
	BoundPredicate

	AsUnbound(Reference) UnboundPredicate
}

BoundUnaryPredicate is a bound predicate expression that has no arguments

type BucketTransform

type BucketTransform struct {
	NumBuckets int
}

BucketTransform transforms values into a bucket partition value. It is parameterized by a number of buckets. Bucket partition transforms use a 32-bit hash of the source value to produce a positive value by mod the bucket number.

func (BucketTransform) Apply

func (t BucketTransform) Apply(value Optional[Literal]) Optional[Literal]

func (BucketTransform) Equals

func (t BucketTransform) Equals(other Transform) bool

func (BucketTransform) MarshalText

func (t BucketTransform) MarshalText() ([]byte, error)

func (BucketTransform) Project

func (t BucketTransform) Project(name string, pred BoundPredicate) (UnboundPredicate, error)

func (BucketTransform) ResultType

func (BucketTransform) ResultType(Type) Type

func (BucketTransform) String

func (t BucketTransform) String() string

func (BucketTransform) Transformer

func (t BucketTransform) Transformer(src Type) func(any) Optional[int32]

type Comparator

type Comparator[T LiteralType] func(v1, v2 T) int

Comparator is a comparison function for specific literal types:

returns 0 if v1 == v2
returns <0 if v1 < v2
returns >0 if v1 > v2

type DataFile

type DataFile interface {
	// ContentType is the type of the content stored by the data file,
	// either Data, Equality deletes, or Position deletes. All v1 files
	// are Data files.
	ContentType() ManifestEntryContent
	// FilePath is the full URI for the file, complete with FS scheme.
	FilePath() string
	// FileFormat is the format of the data file, AVRO, Orc, or Parquet.
	FileFormat() FileFormat
	// Partition returns a mapping of field name to partition value for
	// each of the partition spec's fields.
	Partition() map[string]any
	// Count returns the number of records in this file.
	Count() int64
	// FileSizeBytes is the total file size in bytes.
	FileSizeBytes() int64
	// ColumnSizes is a mapping from column id to the total size on disk
	// of all regions that store the column. Does not include bytes
	// necessary to read other columns, like footers. Map will be nil for
	// row-oriented formats (avro).
	ColumnSizes() map[int]int64
	// ValueCounts is a mapping from column id to the number of values
	// in the column, including null and NaN values.
	ValueCounts() map[int]int64
	// NullValueCounts is a mapping from column id to the number of
	// null values in the column.
	NullValueCounts() map[int]int64
	// NaNValueCounts is a mapping from column id to the number of NaN
	// values in the column.
	NaNValueCounts() map[int]int64
	// DistictValueCounts is a mapping from column id to the number of
	// distinct values in the column. Distinct counts must be derived
	// using values in the file by counting or using sketches, but not
	// using methods like merging existing distinct counts.
	DistinctValueCounts() map[int]int64
	// LowerBoundValues is a mapping from column id to the lower bounded
	// value of the column, serialized as binary. Each value in the column
	// must be less than or requal to all non-null, non-NaN values in the
	// column for the file.
	LowerBoundValues() map[int][]byte
	// UpperBoundValues is a mapping from column id to the upper bounded
	// value of the column, serialized as binary. Each value in the column
	// must be greater than or equal to all non-null, non-NaN values in
	// the column for the file.
	UpperBoundValues() map[int][]byte
	// KeyMetadata is implementation-specific key metadata for encryption.
	KeyMetadata() []byte
	// SplitOffsets are the split offsets for the data file. For example,
	// all row group offsets in a Parquet file. Must be sorted ascending.
	SplitOffsets() []int64
	// EqualityFieldIDs are used to determine row equality in equality
	// delete files. It is required when the content type is
	// EntryContentEqDeletes.
	EqualityFieldIDs() []int
	// SortOrderID returns the id representing the sort order for this
	// file, or nil if there is no sort order.
	SortOrderID() *int
}

DataFile is the interface for reading the information about a given data file indicated by an entry in a manifest list.

type Date

type Date int32

func (Date) ToTime

func (d Date) ToTime() time.Time

type DateLiteral

type DateLiteral Date

func (DateLiteral) Comparator

func (DateLiteral) Comparator() Comparator[Date]

func (DateLiteral) Decrement

func (d DateLiteral) Decrement() Literal

func (DateLiteral) Equals

func (d DateLiteral) Equals(other Literal) bool

func (DateLiteral) Increment

func (d DateLiteral) Increment() Literal

func (DateLiteral) MarshalBinary

func (d DateLiteral) MarshalBinary() (data []byte, err error)

func (DateLiteral) String

func (d DateLiteral) String() string

func (DateLiteral) To

func (d DateLiteral) To(t Type) (Literal, error)

func (DateLiteral) Type

func (d DateLiteral) Type() Type

func (*DateLiteral) UnmarshalBinary

func (d *DateLiteral) UnmarshalBinary(data []byte) error

func (DateLiteral) Value

func (d DateLiteral) Value() Date

type DateType

type DateType struct{}

DateType represents a calendar date without a timezone or time, represented as a 32-bit integer denoting the number of days since the unix epoch.

func (DateType) Equals

func (DateType) Equals(other Type) bool

func (DateType) String

func (DateType) String() string

func (DateType) Type

func (DateType) Type() string

type DayTransform

type DayTransform struct{}

DayTransform transforms a datetime value into a date value.

func (DayTransform) Apply

func (DayTransform) Apply(value Optional[Literal]) (out Optional[Literal])

func (DayTransform) Equals

func (DayTransform) Equals(other Transform) bool

func (DayTransform) MarshalText

func (t DayTransform) MarshalText() ([]byte, error)

func (DayTransform) Project

func (t DayTransform) Project(name string, pred BoundPredicate) (UnboundPredicate, error)

func (DayTransform) ResultType

func (DayTransform) ResultType(Type) Type

func (DayTransform) String

func (DayTransform) String() string

func (DayTransform) Transformer

func (DayTransform) Transformer(src Type) (func(any) Optional[int32], error)

type Decimal

type Decimal struct {
	Val   decimal128.Num
	Scale int
}

type DecimalLiteral

type DecimalLiteral Decimal

func (DecimalLiteral) Comparator

func (DecimalLiteral) Comparator() Comparator[Decimal]

func (DecimalLiteral) Decrement

func (d DecimalLiteral) Decrement() Literal

func (DecimalLiteral) Equals

func (d DecimalLiteral) Equals(other Literal) bool

func (DecimalLiteral) Increment

func (d DecimalLiteral) Increment() Literal

func (DecimalLiteral) MarshalBinary

func (d DecimalLiteral) MarshalBinary() (data []byte, err error)

func (DecimalLiteral) String

func (d DecimalLiteral) String() string

func (DecimalLiteral) To

func (d DecimalLiteral) To(t Type) (Literal, error)

func (DecimalLiteral) Type

func (d DecimalLiteral) Type() Type

func (*DecimalLiteral) UnmarshalBinary

func (d *DecimalLiteral) UnmarshalBinary(data []byte) error

func (DecimalLiteral) Value

func (d DecimalLiteral) Value() Decimal

type DecimalType

type DecimalType struct {
	// contains filtered or unexported fields
}

func DecimalTypeOf

func DecimalTypeOf(prec, scale int) DecimalType

func (DecimalType) Equals

func (d DecimalType) Equals(other Type) bool

func (DecimalType) Precision

func (d DecimalType) Precision() int

func (DecimalType) Scale

func (d DecimalType) Scale() int

func (DecimalType) String

func (d DecimalType) String() string

func (DecimalType) Type

func (d DecimalType) Type() string

type FieldSummary

type FieldSummary struct {
	ContainsNull bool    `avro:"contains_null"`
	ContainsNaN  *bool   `avro:"contains_nan"`
	LowerBound   *[]byte `avro:"lower_bound"`
	UpperBound   *[]byte `avro:"upper_bound"`
}

type FileFormat

type FileFormat string

FileFormat defines constants for the format of data files.

const (
	AvroFile    FileFormat = "AVRO"
	OrcFile     FileFormat = "ORC"
	ParquetFile FileFormat = "PARQUET"
)

type FixedLiteral

type FixedLiteral []byte

func (FixedLiteral) Comparator

func (FixedLiteral) Comparator() Comparator[[]byte]

func (FixedLiteral) Equals

func (f FixedLiteral) Equals(other Literal) bool

func (FixedLiteral) MarshalBinary

func (f FixedLiteral) MarshalBinary() (data []byte, err error)

func (FixedLiteral) String

func (f FixedLiteral) String() string

func (FixedLiteral) To

func (f FixedLiteral) To(typ Type) (Literal, error)

func (FixedLiteral) Type

func (f FixedLiteral) Type() Type

func (*FixedLiteral) UnmarshalBinary

func (f *FixedLiteral) UnmarshalBinary(data []byte) error

func (FixedLiteral) Value

func (f FixedLiteral) Value() []byte

type FixedType

type FixedType struct {
	// contains filtered or unexported fields
}

func FixedTypeOf

func FixedTypeOf(n int) FixedType

func (FixedType) Equals

func (f FixedType) Equals(other Type) bool

func (FixedType) Len

func (f FixedType) Len() int

func (FixedType) String

func (f FixedType) String() string

func (FixedType) Type

func (f FixedType) Type() string

type Float32Literal

type Float32Literal float32

func (Float32Literal) Comparator

func (Float32Literal) Comparator() Comparator[float32]

func (Float32Literal) Equals

func (f Float32Literal) Equals(other Literal) bool

func (Float32Literal) MarshalBinary

func (f Float32Literal) MarshalBinary() (data []byte, err error)

func (Float32Literal) String

func (f Float32Literal) String() string

func (Float32Literal) To

func (f Float32Literal) To(t Type) (Literal, error)

func (Float32Literal) Type

func (f Float32Literal) Type() Type

func (*Float32Literal) UnmarshalBinary

func (f *Float32Literal) UnmarshalBinary(data []byte) error

func (Float32Literal) Value

func (f Float32Literal) Value() float32

type Float32Type

type Float32Type struct{}

Float32Type is the "float" type in the iceberg spec.

func (Float32Type) Equals

func (Float32Type) Equals(other Type) bool

func (Float32Type) String

func (Float32Type) String() string

func (Float32Type) Type

func (Float32Type) Type() string

type Float64Literal

type Float64Literal float64

func (Float64Literal) Comparator

func (Float64Literal) Comparator() Comparator[float64]

func (Float64Literal) Equals

func (f Float64Literal) Equals(other Literal) bool

func (Float64Literal) MarshalBinary

func (f Float64Literal) MarshalBinary() (data []byte, err error)

func (Float64Literal) String

func (f Float64Literal) String() string

func (Float64Literal) To

func (f Float64Literal) To(t Type) (Literal, error)

func (Float64Literal) Type

func (f Float64Literal) Type() Type

func (*Float64Literal) UnmarshalBinary

func (f *Float64Literal) UnmarshalBinary(data []byte) error

func (Float64Literal) Value

func (f Float64Literal) Value() float64

type Float64Type

type Float64Type struct{}

Float64Type represents the "double" type of the iceberg spec.

func (Float64Type) Equals

func (Float64Type) Equals(other Type) bool

func (Float64Type) String

func (Float64Type) String() string

func (Float64Type) Type

func (Float64Type) Type() string

type HourTransform

type HourTransform struct{}

HourTransform transforms a datetime value into an hour value.

func (HourTransform) Apply

func (HourTransform) Apply(value Optional[Literal]) (out Optional[Literal])

func (HourTransform) Equals

func (HourTransform) Equals(other Transform) bool

func (HourTransform) MarshalText

func (t HourTransform) MarshalText() ([]byte, error)

func (HourTransform) Project

func (t HourTransform) Project(name string, pred BoundPredicate) (UnboundPredicate, error)

func (HourTransform) ResultType

func (HourTransform) ResultType(Type) Type

func (HourTransform) String

func (HourTransform) String() string

func (HourTransform) Transformer

func (HourTransform) Transformer(src Type) (func(any) Optional[int32], error)

type IdentityTransform

type IdentityTransform struct{}

IdentityTransform uses the identity function, performing no transformation but instead partitioning on the value itself.

func (IdentityTransform) Apply

func (IdentityTransform) Equals

func (IdentityTransform) Equals(other Transform) bool

func (IdentityTransform) MarshalText

func (t IdentityTransform) MarshalText() ([]byte, error)

func (IdentityTransform) Project

func (IdentityTransform) ResultType

func (IdentityTransform) ResultType(t Type) Type

func (IdentityTransform) String

func (IdentityTransform) String() string

type Int32Literal

type Int32Literal int32

func (Int32Literal) Comparator

func (Int32Literal) Comparator() Comparator[int32]

func (Int32Literal) Decrement

func (i Int32Literal) Decrement() Literal

func (Int32Literal) Equals

func (i Int32Literal) Equals(other Literal) bool

func (Int32Literal) Increment

func (i Int32Literal) Increment() Literal

func (Int32Literal) MarshalBinary

func (i Int32Literal) MarshalBinary() (data []byte, err error)

func (Int32Literal) String

func (i Int32Literal) String() string

func (Int32Literal) To

func (i Int32Literal) To(t Type) (Literal, error)

func (Int32Literal) Type

func (i Int32Literal) Type() Type

func (*Int32Literal) UnmarshalBinary

func (i *Int32Literal) UnmarshalBinary(data []byte) error

func (Int32Literal) Value

func (i Int32Literal) Value() int32

type Int32Type

type Int32Type struct{}

Int32Type is the "int"/"integer" type of the iceberg spec.

func (Int32Type) Equals

func (Int32Type) Equals(other Type) bool

func (Int32Type) String

func (Int32Type) String() string

func (Int32Type) Type

func (Int32Type) Type() string

type Int64Literal

type Int64Literal int64

func (Int64Literal) Comparator

func (Int64Literal) Comparator() Comparator[int64]

func (Int64Literal) Decrement

func (i Int64Literal) Decrement() Literal

func (Int64Literal) Equals

func (i Int64Literal) Equals(other Literal) bool

func (Int64Literal) Increment

func (i Int64Literal) Increment() Literal

func (Int64Literal) MarshalBinary

func (i Int64Literal) MarshalBinary() (data []byte, err error)

func (Int64Literal) String

func (i Int64Literal) String() string

func (Int64Literal) To

func (i Int64Literal) To(t Type) (Literal, error)

func (Int64Literal) Type

func (i Int64Literal) Type() Type

func (*Int64Literal) UnmarshalBinary

func (i *Int64Literal) UnmarshalBinary(data []byte) error

func (Int64Literal) Value

func (i Int64Literal) Value() int64

type Int64Type

type Int64Type struct{}

Int64Type is the "long" type of the iceberg spec.

func (Int64Type) Equals

func (Int64Type) Equals(other Type) bool

func (Int64Type) String

func (Int64Type) String() string

func (Int64Type) Type

func (Int64Type) Type() string

type ListType

type ListType struct {
	ElementID       int  `json:"element-id"`
	Element         Type `json:"-"`
	ElementRequired bool `json:"element-required"`
}

func (*ListType) ElementField

func (l *ListType) ElementField() NestedField

func (*ListType) Equals

func (l *ListType) Equals(other Type) bool

func (*ListType) Fields

func (l *ListType) Fields() []NestedField

func (*ListType) MarshalJSON

func (l *ListType) MarshalJSON() ([]byte, error)

func (*ListType) String

func (l *ListType) String() string

func (*ListType) Type

func (*ListType) Type() string

func (*ListType) UnmarshalJSON

func (l *ListType) UnmarshalJSON(b []byte) error

type Literal

type Literal interface {
	fmt.Stringer
	encoding.BinaryMarshaler

	Type() Type
	To(Type) (Literal, error)
	Equals(Literal) bool
}

Literal is a non-null literal value. It can be casted using To and be checked for equality against other literals.

func Float32AboveMaxLiteral

func Float32AboveMaxLiteral() Literal

func Float32BelowMinLiteral

func Float32BelowMinLiteral() Literal

func Float64AboveMaxLiteral

func Float64AboveMaxLiteral() Literal

func Float64BelowMinLiteral

func Float64BelowMinLiteral() Literal

func Int32AboveMaxLiteral

func Int32AboveMaxLiteral() Literal

func Int32BelowMinLiteral

func Int32BelowMinLiteral() Literal

func Int64AboveMaxLiteral

func Int64AboveMaxLiteral() Literal

func Int64BelowMinLiteral

func Int64BelowMinLiteral() Literal

func LiteralFromBytes

func LiteralFromBytes(typ Type, data []byte) (Literal, error)

LiteralFromBytes uses the defined Iceberg spec for how to serialize a value of a the provided type and returns the appropriate Literal value from it.

If you already have a value of the desired Literal type, you could alternatively call UnmarshalBinary on it yourself manually.

This is primarily used for retrieving stat values.

func NewLiteral

func NewLiteral[T LiteralType](val T) Literal

NewLiteral provides a literal based on the type of T

type LiteralType

type LiteralType interface {
	bool | int32 | int64 | float32 | float64 | Date |
		Time | Timestamp | string | []byte | uuid.UUID | Decimal
}

LiteralType is a generic type constraint for the explicit Go types that we allow for literal values. This represents the actual primitive types that exist in Iceberg

type ManifestContent

type ManifestContent int32

ManifestContent indicates the type of data inside of the files described by a manifest. This will indicate whether the data files contain active data or deleted rows.

const (
	ManifestContentData    ManifestContent = 0
	ManifestContentDeletes ManifestContent = 1
)

type ManifestEntry

type ManifestEntry interface {
	// Status returns the type of the file tracked by this entry.
	// Deletes are informational only and not used in scans.
	Status() ManifestEntryStatus
	// SnapshotID is the id where the file was added, or deleted,
	// if null it is inherited from the manifest list.
	SnapshotID() int64
	// SequenceNum returns the data sequence number of the file.
	// If it was null and the status is EntryStatusADDED then it
	// is inherited from the manifest list.
	SequenceNum() int64
	// FileSequenceNum returns the file sequence number indicating
	// when the file was added. If it was null and the status is
	// EntryStatusADDED then it is inherited from the manifest list.
	FileSequenceNum() *int64
	// DataFile provides the information about the data file indicated
	// by this manifest entry.
	DataFile() DataFile
	// contains filtered or unexported methods
}

ManifestEntry is an interface for both v1 and v2 manifest entries.

type ManifestEntryContent

type ManifestEntryContent int8

ManifestEntryContent defines constants for the type of file contents in the file entries. Data, Position based deletes and equality based deletes.

const (
	EntryContentData       ManifestEntryContent = 0
	EntryContentPosDeletes ManifestEntryContent = 1
	EntryContentEqDeletes  ManifestEntryContent = 2
)

func (ManifestEntryContent) String

func (m ManifestEntryContent) String() string

type ManifestEntryStatus

type ManifestEntryStatus int8

ManifestEntryStatus defines constants for the entry status of existing, added or deleted.

const (
	EntryStatusEXISTING ManifestEntryStatus = 0
	EntryStatusADDED    ManifestEntryStatus = 1
	EntryStatusDELETED  ManifestEntryStatus = 2
)

type ManifestFile

type ManifestFile interface {
	// Version returns the version number of this manifest file.
	// It should be 1 or 2.
	Version() int
	// FilePath is the location URI of this manifest file.
	FilePath() string
	// Length is the length in bytes of the manifest file.
	Length() int64
	// PartitionSpecID is the ID of the partition spec used to write
	// this manifest. It must be listed in the table metadata
	// partition-specs.
	PartitionSpecID() int32
	// ManifestContent is the type of files tracked by this manifest,
	// either data or delete files. All v1 manifests track data files.
	ManifestContent() ManifestContent
	// SnapshotID is the ID of the snapshot where this manifest file
	// was added.
	SnapshotID() int64
	// AddedDataFiles returns the number of entries in the manifest that
	// have the status of EntryStatusADDED.
	AddedDataFiles() int32
	// ExistingDataFiles returns the number of entries in the manifest
	// which have the status of EntryStatusEXISTING.
	ExistingDataFiles() int32
	// DeletedDataFiles returns the number of entries in the manifest
	// which have the status of EntryStatusDELETED.
	DeletedDataFiles() int32
	// AddedRows returns the number of rows in all files of the manifest
	// that have status EntryStatusADDED.
	AddedRows() int64
	// ExistingRows returns the number of rows in all files of the manifest
	// which have status EntryStatusEXISTING.
	ExistingRows() int64
	// DeletedRows returns the number of rows in all files of the manifest
	// which have status EntryStatusDELETED.
	DeletedRows() int64
	// SequenceNum returns the sequence number when this manifest was
	// added to the table. Will be 0 for v1 manifest lists.
	SequenceNum() int64
	// MinSequenceNum is the minimum data sequence number of all live data
	// or delete files in the manifest. Will be 0 for v1 manifest lists.
	MinSequenceNum() int64
	// KeyMetadata returns implementation-specific key metadata for encryption
	// if it exists in the manifest list.
	KeyMetadata() []byte
	// Partitions returns a list of field summaries for each partition
	// field in the spec. Each field in the list corresponds to a field in
	// the manifest file's partition spec.
	Partitions() []FieldSummary

	// HasAddedFiles returns true if AddedDataFiles > 0 or if it was null.
	HasAddedFiles() bool
	// HasExistingFiles returns true if ExistingDataFiles > 0 or if it was null.
	HasExistingFiles() bool
	// FetchEntries reads the manifest list file to fetch the list of
	// manifest entries using the provided file system IO interface.
	// If discardDeleted is true, entries for files containing deleted rows
	// will be skipped.
	FetchEntries(fs iceio.IO, discardDeleted bool) ([]ManifestEntry, error)
}

ManifestFile is the interface which covers both V1 and V2 manifest files.

func ReadManifestList

func ReadManifestList(in io.Reader) ([]ManifestFile, error)

ReadManifestList reads in an avro manifest list file and returns a slice of manifest files or an error if one is encountered.

type ManifestV1Builder

type ManifestV1Builder struct {
	// contains filtered or unexported fields
}

ManifestV1Builder is a helper for building a V1 manifest file struct which will conform to the ManifestFile interface.

func NewManifestV1Builder

func NewManifestV1Builder(path string, length int64, partitionSpecID int32, addedSnapshotID int64) *ManifestV1Builder

NewManifestV1Builder is passed all of the required fields and then allows all of the optional fields to be set by calling the corresponding methods before calling ManifestV1Builder.Build to construct the object.

func (*ManifestV1Builder) AddedFiles

func (b *ManifestV1Builder) AddedFiles(cnt int32) *ManifestV1Builder

func (*ManifestV1Builder) AddedRows

func (b *ManifestV1Builder) AddedRows(cnt int64) *ManifestV1Builder

func (*ManifestV1Builder) Build

func (b *ManifestV1Builder) Build() ManifestFile

Build returns the constructed manifest file, after calling Build this builder should not be used further as we avoid copying by just returning a pointer to the constructed manifest file. Further calls to the modifier methods after calling build would modify the constructed ManifestFile.

func (*ManifestV1Builder) DeletedFiles

func (b *ManifestV1Builder) DeletedFiles(cnt int32) *ManifestV1Builder

func (*ManifestV1Builder) DeletedRows

func (b *ManifestV1Builder) DeletedRows(cnt int64) *ManifestV1Builder

func (*ManifestV1Builder) ExistingFiles

func (b *ManifestV1Builder) ExistingFiles(cnt int32) *ManifestV1Builder

func (*ManifestV1Builder) ExistingRows

func (b *ManifestV1Builder) ExistingRows(cnt int64) *ManifestV1Builder

func (*ManifestV1Builder) KeyMetadata

func (b *ManifestV1Builder) KeyMetadata(km []byte) *ManifestV1Builder

func (*ManifestV1Builder) Partitions

func (b *ManifestV1Builder) Partitions(p []FieldSummary) *ManifestV1Builder

type ManifestV2Builder

type ManifestV2Builder struct {
	// contains filtered or unexported fields
}

ManifestV2Builder is a helper for building a V2 manifest file struct which will conform to the ManifestFile interface.

func NewManifestV2Builder

func NewManifestV2Builder(path string, length int64, partitionSpecID int32, content ManifestContent, addedSnapshotID int64) *ManifestV2Builder

NewManifestV2Builder is constructed with the primary fields, with the remaining fields set to their zero value unless modified by calling the corresponding methods of the builder. Then calling ManifestV2Builder.Build to retrieve the constructed ManifestFile.

func (*ManifestV2Builder) AddedFiles

func (b *ManifestV2Builder) AddedFiles(cnt int32) *ManifestV2Builder

func (*ManifestV2Builder) AddedRows

func (b *ManifestV2Builder) AddedRows(cnt int64) *ManifestV2Builder

func (*ManifestV2Builder) Build

func (b *ManifestV2Builder) Build() ManifestFile

Build returns the constructed manifest file, after calling Build this builder should not be used further as we avoid copying by just returning a pointer to the constructed manifest file. Further calls to the modifier methods after calling build would modify the constructed ManifestFile.

func (*ManifestV2Builder) DeletedFiles

func (b *ManifestV2Builder) DeletedFiles(cnt int32) *ManifestV2Builder

func (*ManifestV2Builder) DeletedRows

func (b *ManifestV2Builder) DeletedRows(cnt int64) *ManifestV2Builder

func (*ManifestV2Builder) ExistingFiles

func (b *ManifestV2Builder) ExistingFiles(cnt int32) *ManifestV2Builder

func (*ManifestV2Builder) ExistingRows

func (b *ManifestV2Builder) ExistingRows(cnt int64) *ManifestV2Builder

func (*ManifestV2Builder) KeyMetadata

func (b *ManifestV2Builder) KeyMetadata(km []byte) *ManifestV2Builder

func (*ManifestV2Builder) Partitions

func (b *ManifestV2Builder) Partitions(p []FieldSummary) *ManifestV2Builder

func (*ManifestV2Builder) SequenceNum

func (b *ManifestV2Builder) SequenceNum(num, minSeqNum int64) *ManifestV2Builder

type MapType

type MapType struct {
	KeyID         int  `json:"key-id"`
	KeyType       Type `json:"-"`
	ValueID       int  `json:"value-id"`
	ValueType     Type `json:"-"`
	ValueRequired bool `json:"value-required"`
}

func (*MapType) Equals

func (m *MapType) Equals(other Type) bool

func (*MapType) Fields

func (m *MapType) Fields() []NestedField

func (*MapType) KeyField

func (m *MapType) KeyField() NestedField

func (*MapType) MarshalJSON

func (m *MapType) MarshalJSON() ([]byte, error)

func (*MapType) String

func (m *MapType) String() string

func (*MapType) Type

func (*MapType) Type() string

func (*MapType) UnmarshalJSON

func (m *MapType) UnmarshalJSON(b []byte) error

func (*MapType) ValueField

func (m *MapType) ValueField() NestedField

type MonthTransform

type MonthTransform struct{}

MonthTransform transforms a datetime value into a month value.

func (MonthTransform) Apply

func (MonthTransform) Apply(value Optional[Literal]) (out Optional[Literal])

func (MonthTransform) Equals

func (MonthTransform) Equals(other Transform) bool

func (MonthTransform) MarshalText

func (t MonthTransform) MarshalText() ([]byte, error)

func (MonthTransform) Project

func (t MonthTransform) Project(name string, pred BoundPredicate) (UnboundPredicate, error)

func (MonthTransform) ResultType

func (MonthTransform) ResultType(Type) Type

func (MonthTransform) String

func (MonthTransform) String() string

func (MonthTransform) Transformer

func (MonthTransform) Transformer(src Type) (func(any) Optional[int32], error)

type NestedField

type NestedField struct {
	Type `json:"-"`

	ID             int    `json:"id"`
	Name           string `json:"name"`
	Required       bool   `json:"required"`
	Doc            string `json:"doc,omitempty"`
	InitialDefault any    `json:"initial-default,omitempty"`
	WriteDefault   any    `json:"write-default,omitempty"`
}

func (*NestedField) Equals

func (n *NestedField) Equals(other NestedField) bool

func (NestedField) MarshalJSON

func (n NestedField) MarshalJSON() ([]byte, error)

func (NestedField) String

func (n NestedField) String() string

func (*NestedField) UnmarshalJSON

func (n *NestedField) UnmarshalJSON(b []byte) error

type NestedType

type NestedType interface {
	Type
	Fields() []NestedField
}

NestedType is an interface that allows access to the child fields of a nested type such as a list/struct/map type.

type NotExpr

type NotExpr struct {
	// contains filtered or unexported fields
}

func (NotExpr) Equals

func (n NotExpr) Equals(other BooleanExpression) bool

func (NotExpr) Negate

func (n NotExpr) Negate() BooleanExpression

func (NotExpr) Op

func (NotExpr) Op() Operation

func (NotExpr) String

func (n NotExpr) String() string

type NumericLiteral

type NumericLiteral interface {
	Literal
	Increment() Literal
	Decrement() Literal
}

type Operation

type Operation int

Operation is an enum used for constants to define what operation a given expression or predicate is going to execute.

const (
	OpTrue  Operation = iota // True
	OpFalse                  // False
	// unary ops
	OpIsNull  // IsNull
	OpNotNull // NotNull
	OpIsNan   // IsNaN
	OpNotNan  // NotNaN
	// literal ops
	OpLT            // LessThan
	OpLTEQ          // LessThanEqual
	OpGT            // GreaterThan
	OpGTEQ          // GreaterThanEqual
	OpEQ            // Equal
	OpNEQ           // NotEqual
	OpStartsWith    // StartsWith
	OpNotStartsWith // NotStartsWith
	// set ops
	OpIn    // In
	OpNotIn // NotIn
	// boolean ops
	OpNot // Not
	OpAnd // And
	OpOr  // Or
)

func (Operation) FlipLR

func (op Operation) FlipLR() Operation

FlipLR returns the correct operation to use if the left and right operands are flipped.

func (Operation) Negate

func (op Operation) Negate() Operation

Negate returns the inverse operation for a given op

func (Operation) String

func (i Operation) String() string

type Optional

type Optional[T any] struct {
	Val   T
	Valid bool
}

Optional represents a typed value that could be null

type OrExpr

type OrExpr struct {
	// contains filtered or unexported fields
}

func (OrExpr) Equals

func (o OrExpr) Equals(other BooleanExpression) bool

func (OrExpr) Negate

func (o OrExpr) Negate() BooleanExpression

func (OrExpr) Op

func (OrExpr) Op() Operation

func (OrExpr) String

func (o OrExpr) String() string

type PartitionField

type PartitionField struct {
	// SourceID is the source column id of the table's schema
	SourceID int `json:"source-id"`
	// FieldID is the partition field id across all the table partition specs
	FieldID int `json:"field-id"`
	// Name is the name of the partition field itself
	Name string `json:"name"`
	// Transform is the transform used to produce the partition value
	Transform Transform `json:"transform"`
}

PartitionField represents how one partition value is derived from the source column by transformation.

func (*PartitionField) String

func (p *PartitionField) String() string

func (*PartitionField) UnmarshalJSON

func (p *PartitionField) UnmarshalJSON(b []byte) error

type PartitionSpec

type PartitionSpec struct {
	// contains filtered or unexported fields
}

PartitionSpec captures the transformation from table data to partition values

func NewPartitionSpec

func NewPartitionSpec(fields ...PartitionField) PartitionSpec

func NewPartitionSpecID

func NewPartitionSpecID(id int, fields ...PartitionField) PartitionSpec

func (*PartitionSpec) CompatibleWith

func (ps *PartitionSpec) CompatibleWith(other *PartitionSpec) bool

CompatibleWith returns true if this partition spec is considered compatible with the passed in partition spec. This means that the two specs have equivalent field lists regardless of the spec id.

func (PartitionSpec) Equals

func (ps PartitionSpec) Equals(other PartitionSpec) bool

Equals returns true iff the field lists are the same AND the spec id is the same between this partition spec and the provided one.

func (*PartitionSpec) Field

func (ps *PartitionSpec) Field(i int) PartitionField

func (*PartitionSpec) FieldsBySourceID

func (ps *PartitionSpec) FieldsBySourceID(fieldID int) []PartitionField

func (*PartitionSpec) ID

func (ps *PartitionSpec) ID() int

func (*PartitionSpec) IsUnpartitioned

func (ps *PartitionSpec) IsUnpartitioned() bool

func (*PartitionSpec) LastAssignedFieldID

func (ps *PartitionSpec) LastAssignedFieldID() int

func (PartitionSpec) MarshalJSON

func (ps PartitionSpec) MarshalJSON() ([]byte, error)

func (*PartitionSpec) NumFields

func (ps *PartitionSpec) NumFields() int

func (*PartitionSpec) PartitionType

func (ps *PartitionSpec) PartitionType(schema *Schema) *StructType

PartitionType produces a struct of the partition spec.

The partition fields should be optional:

  • All partition transforms are required to produce null if the input value is null. This can happen when the source column is optional.
  • Partition fields may be added later, in which case not all files would have the result field and it may be null.

There is a case where we can guarantee that a partition field in the first and only parittion spec that uses a required source column will never be null, but it doesn't seem worth tracking this case.

func (PartitionSpec) String

func (ps PartitionSpec) String() string

func (*PartitionSpec) UnmarshalJSON

func (ps *PartitionSpec) UnmarshalJSON(b []byte) error

type PartnerAccessor

type PartnerAccessor[P any] interface {
	SchemaPartner(P) P
	FieldPartner(partnerStruct P, fieldID int, fieldName string) P
	ListElementPartner(P) P
	MapKeyPartner(P) P
	MapValuePartner(P) P
}

type PrimitiveType

type PrimitiveType interface {
	Type
	// contains filtered or unexported methods
}

type Properties

type Properties map[string]string

type Reference

type Reference string

Reference is a field name not yet bound to a particular field in a schema

func (Reference) Bind

func (r Reference) Bind(s *Schema, caseSensitive bool) (BoundTerm, error)

func (Reference) Equals

func (r Reference) Equals(other UnboundTerm) bool

func (Reference) String

func (r Reference) String() string

type Schema

type Schema struct {
	ID                 int   `json:"schema-id"`
	IdentifierFieldIDs []int `json:"identifier-field-ids"`
	// contains filtered or unexported fields
}

Schema is an Iceberg table schema, represented as a struct with multiple fields. The fields are only exported via accessor methods rather than exposing the slice directly in order to ensure a schema as immutable.

func NewSchema

func NewSchema(id int, fields ...NestedField) *Schema

NewSchema constructs a new schema with the provided ID and list of fields.

func NewSchemaWithIdentifiers

func NewSchemaWithIdentifiers(id int, identifierIDs []int, fields ...NestedField) *Schema

NewSchemaWithIdentifiers constructs a new schema with the provided ID and fields, along with a slice of field IDs to be listed as identifier fields.

func PruneColumns

func PruneColumns(schema *Schema, selected map[int]Void, selectFullTypes bool) (*Schema, error)

PruneColumns visits a schema pruning any columns which do not exist in the provided selected set. Parent fields of a selected child will be retained.

func (*Schema) AsStruct

func (s *Schema) AsStruct() StructType

AsStruct returns a Struct with the same fields as the schema which can then be used as a Type.

func (*Schema) Equals

func (s *Schema) Equals(other *Schema) bool

Equals compares the fields and identifierIDs, but does not compare the schema ID itself.

func (*Schema) Field

func (s *Schema) Field(i int) NestedField

func (*Schema) FieldHasOptionalParent

func (s *Schema) FieldHasOptionalParent(id int) bool

func (*Schema) Fields

func (s *Schema) Fields() []NestedField

func (*Schema) FindColumnName

func (s *Schema) FindColumnName(fieldID int) (string, bool)

FindColumnName returns the name of the column identified by the passed in field id. The second return value reports whether or not the field id was found in the schema.

func (*Schema) FindFieldByID

func (s *Schema) FindFieldByID(id int) (NestedField, bool)

FindFieldByID is like *Schema.FindColumnName, but returns the whole field rather than just the field name.

func (*Schema) FindFieldByName

func (s *Schema) FindFieldByName(name string) (NestedField, bool)

FindFieldByName returns the field identified by the name given, the second return value will be false if no field by this name is found.

Note: This search is done in a case sensitive manner. To perform a case insensitive search, use *Schema.FindFieldByNameCaseInsensitive.

func (*Schema) FindFieldByNameCaseInsensitive

func (s *Schema) FindFieldByNameCaseInsensitive(name string) (NestedField, bool)

FindFieldByNameCaseInsensitive is like *Schema.FindFieldByName, but performs a case insensitive search.

func (*Schema) FindTypeByID

func (s *Schema) FindTypeByID(id int) (Type, bool)

FindTypeByID is like *Schema.FindFieldByID, but returns only the data type of the field.

func (*Schema) FindTypeByName

func (s *Schema) FindTypeByName(name string) (Type, bool)

FindTypeByName is a convenience function for calling *Schema.FindFieldByName, and then returning just the type.

func (*Schema) FindTypeByNameCaseInsensitive

func (s *Schema) FindTypeByNameCaseInsensitive(name string) (Type, bool)

FindTypeByNameCaseInsensitive is like *Schema.FindTypeByName but performs a case insensitive search.

func (*Schema) HighestFieldID

func (s *Schema) HighestFieldID() int

HighestFieldID returns the value of the numerically highest field ID in this schema.

func (*Schema) MarshalJSON

func (s *Schema) MarshalJSON() ([]byte, error)

func (*Schema) NumFields

func (s *Schema) NumFields() int

func (*Schema) Select

func (s *Schema) Select(caseSensitive bool, names ...string) (*Schema, error)

Select creates a new schema with just the fields identified by name passed in the order they are provided. If caseSensitive is false, then fields will be identified by case insensitive search.

An error is returned if a requested name cannot be found.

func (*Schema) String

func (s *Schema) String() string

func (*Schema) Type

func (s *Schema) Type() string

func (*Schema) UnmarshalJSON

func (s *Schema) UnmarshalJSON(b []byte) error

type SchemaVisitor

type SchemaVisitor[T any] interface {
	Schema(schema *Schema, structResult T) T
	Struct(st StructType, fieldResults []T) T
	Field(field NestedField, fieldResult T) T
	List(list ListType, elemResult T) T
	Map(mapType MapType, keyResult, valueResult T) T
	Primitive(p PrimitiveType) T
}

SchemaVisitor is an interface that can be implemented to allow for easy traversal and processing of a schema.

A SchemaVisitor can also optionally implement the Before/After Field, ListElement, MapKey, or MapValue interfaces to allow them to get called at the appropriate points within schema traversal.

type SchemaWithPartnerVisitor

type SchemaWithPartnerVisitor[T, P any] interface {
	Schema(sc *Schema, schemaPartner P, structResult T) T
	Struct(st StructType, structPartner P, fieldResults []T) T
	Field(field NestedField, fieldPartner P, fieldResult T) T
	List(l ListType, listPartner P, elemResult T) T
	Map(m MapType, mapPartner P, keyResult, valResult T) T
	Primitive(p PrimitiveType, primitivePartner P) T
}

type Set

type Set[E any] interface {
	Add(...E)
	Contains(E) bool
	Members() []E
	Equals(Set[E]) bool
	Len() int
	All(func(E) bool) bool
}

type StringLiteral

type StringLiteral string

func (StringLiteral) Comparator

func (StringLiteral) Comparator() Comparator[string]

func (StringLiteral) Equals

func (s StringLiteral) Equals(other Literal) bool

func (StringLiteral) MarshalBinary

func (s StringLiteral) MarshalBinary() (data []byte, err error)

func (StringLiteral) String

func (s StringLiteral) String() string

func (StringLiteral) To

func (s StringLiteral) To(typ Type) (Literal, error)

func (StringLiteral) Type

func (s StringLiteral) Type() Type

func (*StringLiteral) UnmarshalBinary

func (s *StringLiteral) UnmarshalBinary(data []byte) error

func (StringLiteral) Value

func (s StringLiteral) Value() string

type StringType

type StringType struct{}

func (StringType) Equals

func (StringType) Equals(other Type) bool

func (StringType) String

func (StringType) String() string

func (StringType) Type

func (StringType) Type() string

type StructType

type StructType struct {
	FieldList []NestedField `json:"fields"`
}

func (*StructType) Equals

func (s *StructType) Equals(other Type) bool

func (*StructType) Fields

func (s *StructType) Fields() []NestedField

func (*StructType) MarshalJSON

func (s *StructType) MarshalJSON() ([]byte, error)

func (*StructType) String

func (s *StructType) String() string

func (*StructType) Type

func (*StructType) Type() string

type Term

type Term interface {
	fmt.Stringer
	// contains filtered or unexported methods
}

A Term is a simple expression that evaluates to a value

type Time

type Time int64

type TimeLiteral

type TimeLiteral Time

func (TimeLiteral) Comparator

func (TimeLiteral) Comparator() Comparator[Time]

func (TimeLiteral) Equals

func (t TimeLiteral) Equals(other Literal) bool

func (TimeLiteral) MarshalBinary

func (t TimeLiteral) MarshalBinary() (data []byte, err error)

func (TimeLiteral) String

func (t TimeLiteral) String() string

func (TimeLiteral) To

func (t TimeLiteral) To(typ Type) (Literal, error)

func (TimeLiteral) Type

func (t TimeLiteral) Type() Type

func (*TimeLiteral) UnmarshalBinary

func (t *TimeLiteral) UnmarshalBinary(data []byte) error

func (TimeLiteral) Value

func (t TimeLiteral) Value() Time

type TimeType

type TimeType struct{}

TimeType represents a number of microseconds since midnight.

func (TimeType) Equals

func (TimeType) Equals(other Type) bool

func (TimeType) String

func (TimeType) String() string

func (TimeType) Type

func (TimeType) Type() string

type Timestamp

type Timestamp int64

func (Timestamp) ToDate

func (t Timestamp) ToDate() Date

func (Timestamp) ToTime

func (t Timestamp) ToTime() time.Time

type TimestampLiteral

type TimestampLiteral Timestamp

func (TimestampLiteral) Comparator

func (TimestampLiteral) Comparator() Comparator[Timestamp]

func (TimestampLiteral) Decrement

func (t TimestampLiteral) Decrement() Literal

func (TimestampLiteral) Equals

func (t TimestampLiteral) Equals(other Literal) bool

func (TimestampLiteral) Increment

func (t TimestampLiteral) Increment() Literal

func (TimestampLiteral) MarshalBinary

func (t TimestampLiteral) MarshalBinary() (data []byte, err error)

func (TimestampLiteral) String

func (t TimestampLiteral) String() string

func (TimestampLiteral) To

func (t TimestampLiteral) To(typ Type) (Literal, error)

func (TimestampLiteral) Type

func (t TimestampLiteral) Type() Type

func (*TimestampLiteral) UnmarshalBinary

func (t *TimestampLiteral) UnmarshalBinary(data []byte) error

func (TimestampLiteral) Value

func (t TimestampLiteral) Value() Timestamp

type TimestampType

type TimestampType struct{}

TimestampType represents a number of microseconds since the unix epoch without regard for timezone.

func (TimestampType) Equals

func (TimestampType) Equals(other Type) bool

func (TimestampType) String

func (TimestampType) String() string

func (TimestampType) Type

func (TimestampType) Type() string

type TimestampTzType

type TimestampTzType struct{}

TimestampTzType represents a timestamp stored as UTC representing the number of microseconds since the unix epoch.

func (TimestampTzType) Equals

func (TimestampTzType) Equals(other Type) bool

func (TimestampTzType) String

func (TimestampTzType) String() string

func (TimestampTzType) Type

func (TimestampTzType) Type() string

type Transform

type Transform interface {
	fmt.Stringer
	encoding.TextMarshaler
	ResultType(t Type) Type
	Equals(Transform) bool
	Apply(Optional[Literal]) Optional[Literal]
	Project(name string, pred BoundPredicate) (UnboundPredicate, error)
}

Transform is an interface for the various Transformation types in partition specs. Currently, they do not yet provide actual transformation functions or implementation. That will come later as data reading gets implemented.

func ParseTransform

func ParseTransform(s string) (Transform, error)

ParseTransform takes the string representation of a transform as defined in the iceberg spec, and produces the appropriate Transform object or an error if the string is not a valid transform string.

type TruncateTransform

type TruncateTransform struct {
	Width int
}

TruncateTransform is a transformation for truncating a value to a specified width.

func (TruncateTransform) Apply

func (t TruncateTransform) Apply(value Optional[Literal]) (out Optional[Literal])

func (TruncateTransform) Equals

func (t TruncateTransform) Equals(other Transform) bool

func (TruncateTransform) MarshalText

func (t TruncateTransform) MarshalText() ([]byte, error)

func (TruncateTransform) Project

func (TruncateTransform) ResultType

func (TruncateTransform) ResultType(t Type) Type

func (TruncateTransform) String

func (t TruncateTransform) String() string

func (TruncateTransform) Transformer

func (t TruncateTransform) Transformer(src Type) (func(any) any, error)

type Type

type Type interface {
	fmt.Stringer
	Type() string
	Equals(Type) bool
}

Type is an interface representing any of the available iceberg types, such as primitives (int32/int64/etc.) or nested types (list/struct/map).

type TypedLiteral

type TypedLiteral[T LiteralType] interface {
	Literal

	Value() T
	Comparator() Comparator[T]
}

TypedLiteral is a generic interface for Literals so that you can retrieve the value. This is based on the physical representative type, which means that FixedLiteral and BinaryLiteral will both return []byte, etc.

type UUIDLiteral

type UUIDLiteral uuid.UUID

func (UUIDLiteral) Comparator

func (UUIDLiteral) Comparator() Comparator[uuid.UUID]

func (UUIDLiteral) Equals

func (u UUIDLiteral) Equals(other Literal) bool

func (UUIDLiteral) MarshalBinary

func (u UUIDLiteral) MarshalBinary() (data []byte, err error)

func (UUIDLiteral) String

func (u UUIDLiteral) String() string

func (UUIDLiteral) To

func (u UUIDLiteral) To(typ Type) (Literal, error)

func (UUIDLiteral) Type

func (UUIDLiteral) Type() Type

func (*UUIDLiteral) UnmarshalBinary

func (u *UUIDLiteral) UnmarshalBinary(data []byte) error

func (UUIDLiteral) Value

func (u UUIDLiteral) Value() uuid.UUID

type UUIDType

type UUIDType struct{}

func (UUIDType) Equals

func (UUIDType) Equals(other Type) bool

func (UUIDType) String

func (UUIDType) String() string

func (UUIDType) Type

func (UUIDType) Type() string

type UnboundPredicate

type UnboundPredicate interface {
	BooleanExpression

	Term() UnboundTerm
	// contains filtered or unexported methods
}

An UnboundPredicate represents a boolean predicate expression which has not yet been bound to a schema. Binding it will produce a BooleanExpression.

BooleanExpression is used for the binding result because we may optimize and return AlwaysTrue / AlwaysFalse in some scenarios during binding which are not considered to be "Bound" as they do not have a bound Term or Reference.

func EqualTo

func EqualTo[T LiteralType](t UnboundTerm, v T) UnboundPredicate

EqualTo is a convenience wrapper for calling LiteralPredicate(OpEQ, t, NewLiteral(v))

Will panic if t is nil

func GreaterThan

func GreaterThan[T LiteralType](t UnboundTerm, v T) UnboundPredicate

GreaterThan is a convenience wrapper for calling LiteralPredicate(OpGT, t, NewLiteral(v))

Will panic if t is nil

func GreaterThanEqual

func GreaterThanEqual[T LiteralType](t UnboundTerm, v T) UnboundPredicate

GreaterThanEqual is a convenience wrapper for calling LiteralPredicate(OpGTEQ, t, NewLiteral(v))

Will panic if t is nil

func IsNaN

IsNaN is a convenience wrapper for calling UnaryPredicate(OpIsNan, t)

Will panic if t is nil

func IsNull

func IsNull(t UnboundTerm) UnboundPredicate

IsNull is a convenience wrapper for calling UnaryPredicate(OpIsNull, t)

Will panic if t is nil

func LessThan

func LessThan[T LiteralType](t UnboundTerm, v T) UnboundPredicate

LessThan is a convenience wrapper for calling LiteralPredicate(OpLT, t, NewLiteral(v))

Will panic if t is nil

func LessThanEqual

func LessThanEqual[T LiteralType](t UnboundTerm, v T) UnboundPredicate

LessThanEqual is a convenience wrapper for calling LiteralPredicate(OpLTEQ, t, NewLiteral(v))

Will panic if t is nil

func LiteralPredicate

func LiteralPredicate(op Operation, t UnboundTerm, lit Literal) UnboundPredicate

LiteralPredicate constructs an unbound predicate for an operation that requires a single literal argument, such as LessThan or StartsWith.

Panics if the operation provided is not a valid Literal operation, if the term is nil or if the literal is nil.

func NotEqualTo

func NotEqualTo[T LiteralType](t UnboundTerm, v T) UnboundPredicate

NotEqualTo is a convenience wrapper for calling LiteralPredicate(OpNEQ, t, NewLiteral(v))

Will panic if t is nil

func NotNaN

func NotNaN(t UnboundTerm) UnboundPredicate

NotNaN is a convenience wrapper for calling UnaryPredicate(OpNotNan, t)

Will panic if t is nil

func NotNull

func NotNull(t UnboundTerm) UnboundPredicate

NotNull is a convenience wrapper for calling UnaryPredicate(OpNotNull, t)

Will panic if t is nil

func NotStartsWith

func NotStartsWith(t UnboundTerm, v string) UnboundPredicate

NotStartsWith is a convenience wrapper for calling LiteralPredicate(OpNotStartsWith, t, NewLiteral(v))

Will panic if t is nil

func StartsWith

func StartsWith(t UnboundTerm, v string) UnboundPredicate

StartsWith is a convenience wrapper for calling LiteralPredicate(OpStartsWith, t, NewLiteral(v))

Will panic if t is nil

func UnaryPredicate

func UnaryPredicate(op Operation, t UnboundTerm) UnboundPredicate

UnaryPredicate creates and returns an unbound predicate for the provided unary operation. Will panic if op is not a unary operation.

type UnboundTerm

type UnboundTerm interface {
	Term

	Equals(UnboundTerm) bool
	Bind(schema *Schema, caseSensitive bool) (BoundTerm, error)
}

UnboundTerm is an expression that evaluates to a value that isn't yet bound to a schema, thus it isn't yet known what the type will be.

type Void

type Void = struct{}

type VoidTransform

type VoidTransform struct{}

VoidTransform is a transformation that always returns nil.

func (VoidTransform) Apply

func (VoidTransform) Equals

func (VoidTransform) Equals(other Transform) bool

func (VoidTransform) MarshalText

func (t VoidTransform) MarshalText() ([]byte, error)

func (VoidTransform) Project

func (VoidTransform) ResultType

func (VoidTransform) ResultType(t Type) Type

func (VoidTransform) String

func (VoidTransform) String() string

type YearTransform

type YearTransform struct{}

YearTransform transforms a datetime value into a year value.

func (YearTransform) Apply

func (YearTransform) Apply(value Optional[Literal]) (out Optional[Literal])

func (YearTransform) Equals

func (YearTransform) Equals(other Transform) bool

func (YearTransform) MarshalText

func (t YearTransform) MarshalText() ([]byte, error)

func (YearTransform) Project

func (t YearTransform) Project(name string, pred BoundPredicate) (UnboundPredicate, error)

func (YearTransform) ResultType

func (YearTransform) ResultType(Type) Type

func (YearTransform) String

func (YearTransform) String() string

func (YearTransform) Transformer

func (YearTransform) Transformer(src Type) (func(any) Optional[int32], error)

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL