parquet

Documentation

Index

• func ApplyCCL(ctx context.Context, path string, cclScript string) error
• func FilterWithCCL(ctx context.Context, path string, filterExpr string) (*insyra.DataTable, error)
• func Read(ctx context.Context, path string, opt ReadOptions) (*insyra.DataTable, error)
• func ReadColumn(ctx context.Context, path string, column string, opt ReadColumnOptions) (*insyra.DataList, error)
• func Stream(ctx context.Context, path string, opt ReadOptions, batchSize int) (<-chan *insyra.DataTable, <-chan error)
• func Write(dt insyra.IDataTable, path string) error
• type ColumnInfo
• type FileInfo
  • func Inspect(path string) (FileInfo, error)
• type ReadColumnOptions
• type ReadOptions
• type RowGroupInfo

Functions

func ApplyCCL

func ApplyCCL(ctx context.Context, path string, cclScript string) error

ApplyCCL applies CCL expressions directly to parquet file in streaming mode. This function operates directly on parquet files without loading into DataTable. Processing is done batch by batch to minimize memory usage. cclScript can contain multiple statements separated by semicolons.

Example: ApplyCCL(ctx, "input.parquet", "NEW('C') = ['A'] + ['B']; ['D'] = ['D'] * 2", CCLFilterOptions{})

The input file will be overwritten.

func FilterWithCCL

func FilterWithCCL(ctx context.Context, path string, filterExpr string) (*insyra.DataTable, error)

FilterWithCCL applies a CCL filter expression to a parquet file and returns filtered results. The filter expression should evaluate to boolean for each row.

Example: FilterWithCCL(ctx, "input.parquet", "(A > 100) && (B == 'active')")

Returns a new DataTable containing only rows that satisfy the filter condition.

Will not modify the original parquet file.

func Read

func Read(ctx context.Context, path string, opt ReadOptions) (*insyra.DataTable, error)

Read: read parquet file into insyra.DataTable at once

func ReadColumn

func ReadColumn(ctx context.Context, path string, column string, opt ReadColumnOptions) (*insyra.DataList, error)

ReadColumn: read a single column's data from Parquet file. Returns insyra.DataList.

func Stream

func Stream(ctx context.Context, path string, opt ReadOptions, batchSize int) (<-chan *insyra.DataTable, <-chan error)

Stream: streaming read parquet file, returning insyra.DataTable batches

func Write

func Write(dt insyra.IDataTable, path string) error

Write: write insyra.DataTable to parquet file

Types

type ColumnInfo

type ColumnInfo struct {
	Name         string
	PhysicalType string
	LogicalType  string
	Repetition   string
}

type FileInfo

type FileInfo struct {
	NumRows      int64
	NumRowGroups int
	Version      string
	CreatedBy    string
	Metadata     map[string]string
	Columns      []ColumnInfo
	RowGroups    []RowGroupInfo
}

func Inspect

func Inspect(path string) (FileInfo, error)

Inspect: inspect parquet file metadata

type ReadColumnOptions

type ReadColumnOptions struct {
	RowGroups []int // empty=all
	MaxValues int64 // 0=no limit; if exceeded, return error to avoid RAM explosion
}

ReadColumnOptions: Only for ReadColumn (to avoid putting individual requirements into ReadOptions)

type ReadOptions

type ReadOptions struct {
	Columns   []string // empty=all
	RowGroups []int    // empty=all
}

ReadOptions: The options for reading Parquet files

type RowGroupInfo

type RowGroupInfo struct {
	NumRows             int64
	TotalByteSize       int64
	TotalCompressedSize int64
}