orc

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 6, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

README

orc-go

Read and write Apache ORC files in Go.

Use orc.Row for named fields. ReadRows delivers rows to a callback one at a time; ReadAll loads every row into memory. The writer supports nested data, compression, indexes, metadata, and encrypted columns.

Built for Go 1.22 or newer.

Install

go get github.com/fallintoplace/orc-go

Quick start

This example writes one person to people.orc, then reads the name back. The first schema item is the root STRUCT. Its Subtypes point to the child types, and FieldNames gives those fields their names.

package main

import (
	"fmt"
	"log"
	"os"

	orc "github.com/fallintoplace/orc-go"
)

func main() {
	file, err := os.Create("people.orc")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	schema := []orc.Type{
		{Kind: orc.TypeKindStruct, Subtypes: []uint32{1, 2}, FieldNames: []string{"id", "name"}},
		{Kind: orc.TypeKindLong},
		{Kind: orc.TypeKindString},
	}

	writer, err := orc.NewWriter(file, schema, orc.WriterOptions{})
	if err != nil {
		log.Fatal(err)
	}
	if err := writer.WriteRow(orc.Row{"id": int64(1), "name": "Ada"}); err != nil {
		log.Fatal(err)
	}
	if err := writer.Close(); err != nil {
		log.Fatal(err)
	}
	if err := file.Close(); err != nil {
		log.Fatal(err)
	}

	reader, err := orc.Open("people.orc")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	rows, err := reader.ReadAll()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(rows[0]["name"]) // Ada
}

Features

  • Standard ORC types, including nested structs, lists, maps, and unions.
  • NONE, ZLIB, SNAPPY, LZO, LZ4, ZSTD, and BROTLI compression.
  • RLEv1 and RLEv2 integer encodings, plus direct and dictionary string data.
  • Row indexes, bloom filters, file and stripe statistics, and user metadata.
  • AES-CTR column encryption with Hadoop key wrapping or a custom key provider.

See the usage guide for read modes, Go value types, writer options, and encryption details.

Check interoperability

Compare files with PyArrow and Apache ORC Java:

./interop/check-writer.sh

See the interoperability guide for prerequisites.

License

Apache License 2.0. See LICENSE.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrInvalidFile is returned when an ORC file's metadata is malformed.
	ErrInvalidFile = errors.New("invalid ORC file")
	// ErrUnsupportedCompression is returned for an ORC compression codec that is not supported.
	ErrUnsupportedCompression = errors.New("unsupported ORC compression")
	// ErrUnsupportedCalendar is returned for an ORC calendar kind that is not supported.
	ErrUnsupportedCalendar = errors.New("unsupported ORC calendar")
)
View Source 
var (
	// ErrUnsupportedType is returned when row reading encounters a type that is
	// not implemented yet.
	ErrUnsupportedType = errors.New("unsupported ORC type")
	// ErrUnsupportedEncoding is returned when row reading encounters a column
	// encoding that is not implemented yet.
	ErrUnsupportedEncoding = errors.New("unsupported ORC column encoding")
)
View Source 
var ErrColumnNotEncrypted = errors.New("ORC column is not encrypted")

ErrColumnNotEncrypted is returned when a column has no encrypted variant.

View Source 
var ErrEncryptedStatisticsUnavailable = errors.New("encrypted ORC statistics are unavailable")

ErrEncryptedStatisticsUnavailable is returned when encrypted statistics are not stored for a column or are incomplete.

View Source 
var ErrKeyUnavailable = errors.New("ORC encryption key is unavailable")

ErrKeyUnavailable tells a reader that a key is not available and the encrypted column's masked values should be used.

View Source 
var ErrUnsupportedEncryption = errors.New("unsupported ORC encryption")

ErrUnsupportedEncryption is returned for encryption metadata or algorithms that this implementation cannot read.

View Source 
var ErrWriterClosed = errors.New("ORC writer is closed")

ErrWriterClosed is returned when a writer operation is called after Close.

Functions

This section is empty.

Types

type BinaryStatistics 

type BinaryStatistics struct {
	Sum    int64
	HasSum bool
}

BinaryStatistics describes binary columns.

type BloomFilter 

type BloomFilter struct {
	NumHashFunctions uint32
	Bitset           []uint64
	UTF8Bitset       []byte
}

BloomFilter is one row group's ORC bloom filter. Bitset contains the original repeated-fixed64 encoding; UTF8Bitset contains the byte encoding introduced by ORC-101. A filter can contain either or both representations.

func (BloomFilter) MightContainBytes 

func (f BloomFilter) MightContainBytes(value []byte) bool

MightContainBytes reports whether a byte sequence may be present in this filter. A true result can be a false positive; a false result is definitive.

func (BloomFilter) MightContainFloat64 

func (f BloomFilter) MightContainFloat64(value float64) bool

MightContainFloat64 reports whether a floating-point value may be present in this filter.

func (BloomFilter) MightContainInt64 

func (f BloomFilter) MightContainInt64(value int64) bool

MightContainInt64 reports whether an integer may be present in this filter.

func (BloomFilter) MightContainString 

func (f BloomFilter) MightContainString(value string) bool

MightContainString reports whether a string may be present in this filter. A true result can be a false positive; a false result is definitive.

type BucketStatistics 

type BucketStatistics struct {
	Counts []uint64
}

BucketStatistics contains the counts for boolean values, in false/true order.

type CalendarKind 

type CalendarKind uint32

CalendarKind identifies the calendar used to encode historical dates and timestamps in an ORC file. 

const (
	// CalendarKindUnknown means that the ORC file does not name a calendar.
	CalendarKindUnknown CalendarKind = iota
	// CalendarKindJulianGregorian uses the Julian calendar before October 1582
	// and the Gregorian calendar from October 15, 1582 onward.
	CalendarKindJulianGregorian
	// CalendarKindProlepticGregorian uses the Gregorian calendar for all dates.
	CalendarKindProlepticGregorian
)

type CollectionStatistics 

type CollectionStatistics struct {
	MinimumChildren    uint64
	HasMinimumChildren bool
	MaximumChildren    uint64
	HasMaximumChildren bool
	TotalChildren      uint64
	HasTotalChildren   bool
}

CollectionStatistics describes the number of children in LIST and MAP values. Has fields distinguish an omitted count from a valid zero count.

type ColumnBloomFilterIndex 

type ColumnBloomFilterIndex struct {
	ColumnID uint32
	// BloomEncoding is 0 for original filters and 1 for UTC timestamp filters.
	// Other values are preserved for forward compatibility.
	BloomEncoding uint32
	Filters       []BloomFilter
}

ColumnBloomFilterIndex contains row-group bloom filters for one ORC type id.

type ColumnEncryptionStatistics 

type ColumnEncryptionStatistics struct {
	File    []ColumnStatistics
	Stripes [][]ColumnStatistics
}

ColumnEncryptionStatistics contains unmasked statistics for an encrypted root field and its type subtree. File and each stripe's statistics are in schema type-id order, starting with the encrypted root field.

type ColumnRowIndex 

type ColumnRowIndex struct {
	ColumnID uint32
	Entries  []RowIndexEntry
}

ColumnRowIndex contains the row groups indexed for one ORC type id.

type ColumnStatistics 

type ColumnStatistics struct {
	// NumberOfValues is the number of non-null values recorded for this column.
	NumberOfValues    uint64
	HasNumberOfValues bool
	// HasNull reports whether the column includes any null values.
	HasNull bool
	// HasNullSet reports whether the file explicitly stores the hasNull field.
	HasNullSet bool
	// BytesOnDisk is the encoded size recorded for this column, when available.
	BytesOnDisk    uint64
	HasBytesOnDisk bool
	// The type-specific statistics below are nil when the file does not include
	// the corresponding statistic.
	Integer    *IntegerStatistics
	Double     *DoubleStatistics
	String     *StringStatistics
	Bucket     *BucketStatistics
	Decimal    *DecimalStatistics
	Date       *DateStatistics
	Binary     *BinaryStatistics
	Timestamp  *TimestampStatistics
	Collection *CollectionStatistics
	Geospatial *GeospatialStatistics
}

ColumnStatistics contains the statistics recorded for one ORC type id. Type-specific fields are nil when the file does not include that statistic.

type CompressionKind 

type CompressionKind uint32

CompressionKind identifies the codec used for ORC file sections and streams. 

const (
	CompressionNone CompressionKind = iota
	CompressionZlib
	CompressionSnappy
	CompressionLZO
	CompressionLZ4
	CompressionZstd
	CompressionBrotli
)

func (CompressionKind) String 

func (kind CompressionKind) String() string

type DateStatistics 

type DateStatistics struct {
	Minimum    int32
	HasMinimum bool
	Maximum    int32
	HasMaximum bool
}

DateStatistics stores the minimum and maximum day counts since 1970-01-01. The calendar used to interpret those counts is reported by Reader.CalendarKind.

type Decimal 

type Decimal string

Decimal is an exact base-10 value. Its string form uses an optional sign and decimal point, without an exponent (for example, Decimal("-12.340")). The named type keeps decimal values distinct from ORC STRING values.

func ParseDecimal 

func ParseDecimal(value string) (Decimal, error)

ParseDecimal validates a decimal string and returns it without changing its spelling or scale.

func (Decimal) String 

func (value Decimal) String() string

String returns the exact decimal text.

type DecimalStatistics 

type DecimalStatistics struct {
	Minimum    string
	HasMinimum bool
	Maximum    string
	HasMaximum bool
	Sum        string
	HasSum     bool
}

DecimalStatistics describes decimal columns using exact ORC decimal text.

type DoubleStatistics 

type DoubleStatistics struct {
	Minimum    float64
	HasMinimum bool
	Maximum    float64
	HasMaximum bool
	Sum        float64
	HasSum     bool
}

DoubleStatistics describes floating-point columns.

type EdgeInterpolationAlgorithm 

type EdgeInterpolationAlgorithm uint32

EdgeInterpolationAlgorithm selects how geography edges are interpreted. 

const (
	// EdgeInterpolationSpherical is the default spherical geodesic algorithm.
	EdgeInterpolationSpherical EdgeInterpolationAlgorithm = iota
	EdgeInterpolationVincenty
	EdgeInterpolationThomas
	EdgeInterpolationAndoyer
	EdgeInterpolationKarney
)

type EncryptionAlgorithm 

type EncryptionAlgorithm uint32

EncryptionAlgorithm selects the AES key size used for ORC column streams. 

const (
	// EncryptionAESCTR128 uses AES-CTR with a 128-bit local key.
	EncryptionAESCTR128 EncryptionAlgorithm = 1
	// EncryptionAESCTR256 uses AES-CTR with a 256-bit local key.
	EncryptionAESCTR256 EncryptionAlgorithm = 2
)

type EncryptionKeyProvider 

type EncryptionKeyProvider interface {
	// Kind identifies the key provider format written to the ORC file.
	Kind() KeyProviderKind
	// GenerateLocalKey creates an AES key and the opaque bytes needed to
	// recover it later.
	GenerateLocalKey(name string, version uint32, algorithm EncryptionAlgorithm) (localKey, encryptedLocalKey []byte, err error)
	// DecryptLocalKey recovers a local AES key from its opaque bytes.
	DecryptLocalKey(name string, version uint32, algorithm EncryptionAlgorithm, encryptedLocalKey []byte) ([]byte, error)
}

EncryptionKeyProvider wraps and unwraps ORC local encryption keys. Use this interface to connect a KMS without adding its SDK to this package. The returned local key must be 16 bytes for AES-CTR-128 or 32 bytes for AES-CTR-256. The encrypted local key is stored in the ORC file as opaque bytes. Kind must be a provider kind defined by ORC. Return ErrKeyUnavailable when a reader cannot access the key.

type EncryptionMask 

type EncryptionMask struct {
	// Name is the mask name.
	Name string
	// Parameters contains the mask's stored parameters in order.
	Parameters []string
	// Columns contains unencrypted root type IDs covered by the mask.
	Columns []uint32
}

EncryptionMask describes a fallback data mask recorded in ORC encryption metadata. Columns contains the unencrypted root column IDs for the mask.

type EncryptionRule 

type EncryptionRule struct {
	Column     string
	KeyName    string
	KeyVersion uint32
	Algorithm  EncryptionAlgorithm
}

EncryptionRule encrypts a root field and its complete type subtree. Column names are root STRUCT field names.

type GeospatialBoundingBox 

type GeospatialBoundingBox struct {
	XMin    float64
	HasXMin bool
	XMax    float64
	HasXMax bool
	YMin    float64
	HasYMin bool
	YMax    float64
	HasYMax bool
	ZMin    float64
	HasZMin bool
	ZMax    float64
	HasZMax bool
	MMin    float64
	HasMMin bool
	MMax    float64
	HasMMax bool
}

GeospatialBoundingBox contains the minimum and maximum coordinate for each axis present in the column's geospatial values.

type GeospatialStatistics 

type GeospatialStatistics struct {
	BoundingBox     *GeospatialBoundingBox
	GeospatialTypes []int32
}

GeospatialStatistics describes the values in a GEOMETRY or GEOGRAPHY column.

type IntegerStatistics 

type IntegerStatistics struct {
	Minimum    int64
	HasMinimum bool
	Maximum    int64
	HasMaximum bool
	Sum        int64
	HasSum     bool
}

IntegerStatistics describes integer, byte, and long columns.

type KeyProvider 

type KeyProvider interface {
	GetKey(name string, version uint32) ([]byte, error)
}

KeyProvider resolves a named and versioned AES master key for Hadoop ORC key wrapping. Use EncryptionKeyProvider for other wrapping formats. The ORC file stores wrapped local keys, never the master key itself.

type KeyProviderFunc 

type KeyProviderFunc func(name string, version uint32) ([]byte, error)

KeyProviderFunc adapts a function to the KeyProvider interface.

func (KeyProviderFunc) GetKey 

func (f KeyProviderFunc) GetKey(name string, version uint32) ([]byte, error)

GetKey resolves a key using f.

type KeyProviderKind 

type KeyProviderKind uint32

KeyProviderKind identifies how the encrypted local keys in an ORC file are wrapped. 

const (
	// KeyProviderKindUnknown is used when the file does not name a provider.
	KeyProviderKindUnknown KeyProviderKind = iota
	// KeyProviderKindHadoop uses Hadoop or Ranger KMS wrapping.
	KeyProviderKindHadoop
	// KeyProviderKindAWS uses AWS KMS wrapping.
	KeyProviderKindAWS
	// KeyProviderKindGCP uses Google Cloud KMS wrapping.
	KeyProviderKindGCP
	// KeyProviderKindAzure uses Azure Key Vault wrapping.
	KeyProviderKindAzure
)

type MapEntry 

type MapEntry struct {
	Key   any
	Value any
}

MapEntry is one key/value pair in an ORC MAP value. A slice preserves entry order and supports key types that cannot be Go map keys.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads ORC file metadata and decodes root values from its stripes.

func NewReader 

func NewReader(source io.ReaderAt, size int64) (*Reader, error)

NewReader reads an ORC file from a random access source. The caller retains ownership of source and is responsible for closing it if needed.

func NewReaderWithOptions 

func NewReaderWithOptions(source io.ReaderAt, size int64, options ReaderOptions) (*Reader, error)

NewReaderWithOptions reads an ORC file from a random access source. The caller retains ownership of source and is responsible for closing it.

func Open 

func Open(path string) (*Reader, error)

Open opens an ORC file from disk.

func OpenWithOptions 

func OpenWithOptions(path string, options ReaderOptions) (*Reader, error)

OpenWithOptions opens a file with reader options.

func (*Reader) BloomFilterIndexes 

func (r *Reader) BloomFilterIndexes() ([]StripeBloomFilterIndexes, error)

BloomFilterIndexes reads BLOOM_FILTER and BLOOM_FILTER_UTF8 streams from each stripe. Encrypted filters are decoded when a key is available; without a key, encrypted columns expose only their unencrypted masked filters.

func (*Reader) CalendarKind 

func (r *Reader) CalendarKind() CalendarKind

CalendarKind reports the calendar used to interpret DATE and TIMESTAMP values. A known footer value takes priority; otherwise the reader uses the configured fallback or Julian/Gregorian by default.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the file when the Reader was created with Open.

func (*Reader) Compression 

func (r *Reader) Compression() CompressionKind

Compression returns the codec declared by the file postscript.

func (*Reader) EncryptionMasks 

func (r *Reader) EncryptionMasks() []EncryptionMask

EncryptionMasks returns the fallback data masks declared in the file. The returned slices do not alias the reader's encryption metadata.

func (*Reader) EncryptionStatistics 

func (r *Reader) EncryptionStatistics(column string) (ColumnEncryptionStatistics, error)

EncryptionStatistics returns the unmasked file and stripe statistics for an encrypted root field. A key provider must be configured on the Reader.

func (*Reader) FileStatistics 

func (r *Reader) FileStatistics() []ColumnStatistics

FileStatistics returns the statistics for each column in type-id order. Values in encrypted columns use the unencrypted masked statistics, when present. Encrypted statistics are not exposed by this API.

func (*Reader) FormatVersion 

func (r *Reader) FormatVersion() []uint32

FormatVersion returns the ORC file version declared by the postscript.

func (*Reader) NumRows 

func (r *Reader) NumRows() uint64

NumRows returns the total number of rows declared by the file footer.

func (*Reader) NumStripes 

func (r *Reader) NumStripes() int

NumStripes returns the number of stripes in the file.

func (*Reader) ReadAll 

func (r *Reader) ReadAll() ([]Row, error)

ReadAll reads all rows into memory. Use ReadRows for large files.

func (*Reader) ReadAllRange 

func (r *Reader) ReadAllRange(start, end uint64) ([]Row, error)

ReadAllRange reads the rows in [start, end) into memory.

func (*Reader) ReadAllRangeWithColumns 

func (r *Reader) ReadAllRangeWithColumns(start, end uint64, names []string) ([]Row, error)

ReadAllRangeWithColumns reads [start, end) with only the named root fields.

func (*Reader) ReadAllValues 

func (r *Reader) ReadAllValues() ([]any, error)

ReadAllValues reads all values of the root type into memory. Use ReadValues for large files.

func (*Reader) ReadAllValuesRange 

func (r *Reader) ReadAllValuesRange(start, end uint64) ([]any, error)

ReadAllValuesRange reads root values from the zero-based range [start, end) into memory.

func (*Reader) ReadAllWithColumns 

func (r *Reader) ReadAllWithColumns(names []string) ([]Row, error)

ReadAllWithColumns reads all rows with only the named root fields. Use ReadRowsWithColumns for large files.

func (*Reader) ReadRowBatches added in v0.2.0 

func (r *Reader) ReadRowBatches(yield func(RowBatch) error) error

ReadRowBatches calls yield with up to 1,024 rows at a time without building a map for each row. Fields are available by name through RowBatch.

func (*Reader) ReadRowBatchesWithColumns added in v0.2.0 

func (r *Reader) ReadRowBatchesWithColumns(names []string, yield func(RowBatch) error) error

ReadRowBatchesWithColumns reads all rows and yields only the named root fields. A nil names slice selects all root fields.

func (*Reader) ReadRows 

func (r *Reader) ReadRows(yield func(Row) error) error

ReadRows calls yield for each root STRUCT value in file order. A null root STRUCT is returned as a nil Row. It keeps decoded values for one stripe in memory at a time. Returning an error from yield stops the read.

func (*Reader) ReadRowsRange 

func (r *Reader) ReadRowsRange(start, end uint64, yield func(Row) error) error

ReadRowsRange calls yield for the rows in [start, end), using zero-based file row numbers. Stripes outside the range are not read.

func (*Reader) ReadRowsRangeValuesWithColumns added in v0.2.0 

func (r *Reader) ReadRowsRangeValuesWithColumns(start, end uint64, names []string, yield func(RowValues) error) error

ReadRowsRangeValuesWithColumns reads [start, end) and yields selected root fields as RowValues. It avoids allocating a map for every row. Fields follow the order in names, and values remain available after the callback returns.

func (*Reader) ReadRowsRangeWithColumns 

func (r *Reader) ReadRowsRangeWithColumns(start, end uint64, names []string, yield func(Row) error) error

ReadRowsRangeWithColumns reads [start, end) and calls yield with only the named root fields. Selecting a nested field's root returns its full value.

func (*Reader) ReadRowsValuesWithColumns added in v0.2.0 

func (r *Reader) ReadRowsValuesWithColumns(names []string, yield func(RowValues) error) error

ReadRowsValuesWithColumns reads all root STRUCT rows and yields only the named fields as RowValues. It avoids allocating a map for every row.

func (*Reader) ReadRowsWithColumns 

func (r *Reader) ReadRowsWithColumns(names []string, yield func(Row) error) error

ReadRowsWithColumns calls yield with only the named root fields. Selecting a nested field's root returns the full nested value for that field.

func (*Reader) ReadValues 

func (r *Reader) ReadValues(yield func(any) error) error

ReadValues calls yield for each value of the root type in file order. It keeps decoded values for one stripe in memory at a time. Root STRUCT values are returned as Row; other values use the Go type for their ORC type.

func (*Reader) ReadValuesRange 

func (r *Reader) ReadValuesRange(start, end uint64, yield func(any) error) error

ReadValuesRange reads root values from the zero-based range [start, end). Stripes outside the range are skipped.

func (*Reader) RowIndexStride 

func (r *Reader) RowIndexStride() uint64

RowIndexStride returns the row group size declared by the file footer.

func (*Reader) RowIndexes 

func (r *Reader) RowIndexes() ([]StripeRowIndexes, error)

RowIndexes reads and decodes the ROW_INDEX streams in each stripe. It does not load column data streams. Encrypted indexes are decoded when a key is available; otherwise only unencrypted masked indexes are returned.

func (*Reader) Schema 

func (r *Reader) Schema() []Type

Schema returns a copy of the file's ORC type tree.

func (*Reader) StripeStatistics 

func (r *Reader) StripeStatistics() [][]ColumnStatistics

StripeStatistics returns the statistics for each stripe, with columns in type-id order. The returned nested slices do not alias reader state.

func (*Reader) Stripes 

func (r *Reader) Stripes() []StripeInfo

Stripes returns a copy of the file's stripe directory.

func (*Reader) UserMetadata 

func (r *Reader) UserMetadata() []UserMetadataItem

UserMetadata returns the application-defined key/value pairs in the file footer. The returned values do not alias the reader's metadata.

func (*Reader) WriterMetadata 

func (r *Reader) WriterMetadata() WriterMetadata

WriterMetadata returns the writer ID, writer version, and software version declared in the file, preserving whether each optional field was present.

type ReaderOptions 

type ReaderOptions struct {
	// DefaultCalendar is used for DATE and TIMESTAMP values when the file has
	// no calendar metadata. Zero defaults to the Julian/Gregorian calendar.
	DefaultCalendar CalendarKind
	// KeyProvider resolves Hadoop master keys used to wrap ORC local keys. If
	// nil and no matching EncryptionKeyProvider is set, encrypted columns use
	// their unencrypted masked variant. When both are set for a Hadoop file,
	// KeyProvider is tried if EncryptionKeyProvider reports the key unavailable.
	KeyProvider KeyProvider
	// EncryptionKeyProvider handles the matching ORC key-provider format. It is
	// tried before KeyProvider and can connect a KMS without adding its SDK.
	EncryptionKeyProvider EncryptionKeyProvider
}

ReaderOptions controls optional reader behavior.

type Row 

type Row map[string]any

Row contains one ORC root-struct value. A nil Row represents a null root STRUCT. Nested STRUCT values also use Row, LIST values use slices, MAP values use []MapEntry, and UNION values use UnionValue. Primitive values use bool for BOOLEAN, the matching integer width for integer columns, time.Time for DATE and timestamps, Decimal for DECIMAL, float32 or float64 for FLOAT or DOUBLE, string for STRING, CHAR, or VARCHAR, and []byte for BINARY, GEOMETRY, or GEOGRAPHY. Geospatial values use ORC's WKB representation. Timestamp values use UTC; TIMESTAMP preserves wall-clock fields while TIMESTAMP_INSTANT preserves the instant. A nil value represents an ORC null.

type RowBatch added in v0.2.0 

type RowBatch struct {
	// contains filtered or unexported fields
}

RowBatch contains consecutive root STRUCT rows in named columns. Each column slice is aligned with row indexes and remains valid after the read callback returns. A null root has nil values in every column; use IsNull to distinguish it from a non-null row with a null field.

func (RowBatch) Column added in v0.2.0 

func (b RowBatch) Column(name string) ([]any, bool)

Column returns the row-aligned values for name. Null root rows contain nil; use IsNull to distinguish them from null fields.

func (RowBatch) Field added in v0.2.0 

func (b RowBatch) Field(row, field int) (string, any, bool)

Field returns the name and value at the field index for row.

func (RowBatch) FieldNames added in v0.2.0 

func (b RowBatch) FieldNames() []string

FieldNames returns the selected field names in batch order.

func (RowBatch) Get added in v0.2.0 

func (b RowBatch) Get(row int, name string) (any, bool)

Get returns a field value by row index and name. The second result is false when the row or field is outside the batch. A null field returns nil, true.

func (RowBatch) IsNull added in v0.2.0 

func (b RowBatch) IsNull(row int) (bool, bool)

IsNull reports whether the root STRUCT at row is null. The second result is false when row is outside the batch.

func (RowBatch) Len added in v0.2.0 

func (b RowBatch) Len() int

Len returns the number of rows in the batch.

type RowIndexEntry 

type RowIndexEntry struct {
	Positions     []uint64
	Statistics    ColumnStatistics
	HasStatistics bool
}

RowIndexEntry contains the stream positions and optional statistics for one row group in an ORC column.

type RowValues added in v0.2.0 

type RowValues struct {
	// contains filtered or unexported fields
}

RowValues contains one row's selected fields without constructing a Row map. Its values remain valid after the callback returns.

func (RowValues) Field added in v0.2.0 

func (r RowValues) Field(index int) (string, any, bool)

Field returns the name and value at the selected-field index.

func (RowValues) Get added in v0.2.0 

func (r RowValues) Get(name string) (any, bool)

Get returns a selected field by name. The second result is false when the field was not selected. It is true when the selected value is nil, including when the row has a null root STRUCT.

func (RowValues) IsNull added in v0.2.0 

func (r RowValues) IsNull() bool

IsNull reports whether the root STRUCT for this row is null.

func (RowValues) Len added in v0.2.0 

func (r RowValues) Len() int

Len returns the number of selected fields in the row.

type StringStatistics 

type StringStatistics struct {
	Minimum       string
	HasMinimum    bool
	Maximum       string
	HasMaximum    bool
	LowerBound    string
	HasLowerBound bool
	UpperBound    string
	HasUpperBound bool
	Sum           int64
	HasSum        bool
}

StringStatistics describes string-like columns. HasMinimum and HasMaximum are false when ORC stores only a truncated bound. Sum is the total UTF-8 byte length of the values, as stored by ORC.

type StripeBloomFilterIndexes 

type StripeBloomFilterIndexes struct {
	Columns []ColumnBloomFilterIndex
}

StripeBloomFilterIndexes contains the bloom filter indexes found in one stripe. Columns are ordered by type id; columns without a bloom-filter stream are omitted.

type StripeInfo 

type StripeInfo struct {
	Offset       uint64
	IndexLength  uint64
	DataLength   uint64
	FooterLength uint64
	NumberOfRows uint64
}

StripeInfo contains the byte ranges and row count for one stripe.

type StripeRowIndexes 

type StripeRowIndexes struct {
	Columns []ColumnRowIndex
}

StripeRowIndexes contains the row indexes found in one stripe. Columns are ordered by type id; columns without a ROW_INDEX stream are omitted.

type TimestampStatistics 

type TimestampStatistics struct {
	Minimum         int64
	HasMinimum      bool
	Maximum         int64
	HasMaximum      bool
	MinimumUTC      int64
	HasMinimumUTC   bool
	MaximumUTC      int64
	HasMaximumUTC   bool
	MinimumNanos    int32
	HasMinimumNanos bool
	MaximumNanos    int32
	HasMaximumNanos bool
}

TimestampStatistics stores timestamp minima and maxima as Unix milliseconds. The UTC fields are present in newer ORC writers and distinguish instant comparisons from wall-clock timestamp comparisons. The nanosecond fields store the final six digits below millisecond precision, as values from 0 to 999999. When a nanosecond field is absent, ORC uses 0 for a minimum and 999999 for a maximum.

type Type 

type Type struct {
	Kind       TypeKind
	Subtypes   []uint32
	FieldNames []string
	// Attributes contains ORC key/value annotations for this type.
	Attributes []TypeAttribute
	// MaxLength limits CHAR and VARCHAR values in Unicode code points.
	MaxLength uint32
	// Precision and Scale may both be zero for legacy variable-scale DECIMAL.
	Precision uint32
	Scale     uint32
	// CRS sets the coordinate reference system for GEOMETRY and GEOGRAPHY.
	// Empty uses the ORC default, OGC:CRS84.
	CRS string
	// EdgeInterpolation sets the GEOGRAPHY edge algorithm. Zero selects
	// EdgeInterpolationSpherical, the ORC default.
	EdgeInterpolation EdgeInterpolationAlgorithm
}

Type describes one node in the ORC type tree. Subtypes refer to indexes in the schema's Types slice; ORC stores the root type at index zero.

type TypeAttribute 

type TypeAttribute struct {
	Key   string
	Value string
}

TypeAttribute is a key/value annotation stored on an ORC type.

type TypeKind 

type TypeKind uint32

TypeKind identifies an ORC column type. 

const (
	TypeKindBoolean TypeKind = iota
	TypeKindByte
	TypeKindShort
	TypeKindInt
	TypeKindLong
	TypeKindFloat
	TypeKindDouble
	TypeKindString
	TypeKindBinary
	TypeKindTimestamp
	TypeKindList
	TypeKindMap
	TypeKindStruct
	TypeKindUnion
	TypeKindDecimal
	TypeKindDate
	TypeKindVarchar
	TypeKindChar
	TypeKindTimestampInstant
	TypeKindGeometry
	TypeKindGeography
)

func (TypeKind) String 

func (kind TypeKind) String() string

type UnionValue 

type UnionValue struct {
	Tag   uint8
	Value any
}

UnionValue stores the selected ORC UNION alternative. Tag is the zero-based index into the union type's Subtypes slice.

type UserMetadataItem 

type UserMetadataItem struct {
	Name  string
	Value []byte
}

UserMetadataItem is an application-defined key and byte value stored in an ORC file footer. Column encryption does not encrypt user metadata.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes ORC files for primitive or nested root values. It buffers values for the current stripe and writes completed stripes to the destination. Close writes the file footer and does not close the destination.

func NewWriter 

func NewWriter(destination io.Writer, schema []Type, options WriterOptions) (*Writer, error)

NewWriter creates a writer for a supported ORC type tree. The root may be any supported scalar or compound type. Use WriteRow for root STRUCT schemas and WriteValue for any root type.

func (*Writer) AddUserMetadata 

func (w *Writer) AddUserMetadata(name string, value []byte) error

AddUserMetadata appends an application-defined key/value pair to the file footer. Both the name and value are copied before the call returns. Column encryption does not encrypt user metadata.

func (*Writer) Close 

func (w *Writer) Close() error

Close writes the final stripe and ORC file footer. Calling Close again returns the result of the first call.

func (*Writer) WriteRow 

func (w *Writer) WriteRow(row Row) error

WriteRow adds one non-null root STRUCT row. Missing fields and nil values are written as ORC nulls. Use WriteValue to write a null root STRUCT. Binary and geospatial byte values are copied before they are buffered.

func (*Writer) WriteValue 

func (w *Writer) WriteValue(value any) error

WriteValue adds one value of the schema's root type. Use this method when the root is not a STRUCT, or when a root STRUCT itself may be null. For a root STRUCT, non-null values may be Row or map[string]any. Binary and geospatial byte values are copied before they are buffered.

type WriterMetadata 

type WriterMetadata struct {
	// ImplementationID is the registered ORC writer ID, when present.
	ImplementationID uint32
	// HasImplementationID reports whether the footer includes a writer ID.
	HasImplementationID bool
	// SoftwareVersion is the writer's descriptive version string, when present.
	SoftwareVersion string
	// HasSoftwareVersion reports whether the footer includes a software version.
	HasSoftwareVersion bool
	// WriterVersion is the writer's version sequence from the postscript.
	WriterVersion uint32
	// HasWriterVersion reports whether the postscript includes a writer version.
	HasWriterVersion bool
}

WriterMetadata identifies the writer declared in an ORC file footer and postscript.

type WriterOptions 

type WriterOptions struct {
	// Compression selects NONE, ZLIB, SNAPPY, LZO, LZ4, ZSTD, or BROTLI.
	Compression CompressionKind
	// CompressionBlockSize sets the maximum uncompressed chunk size. Zero uses
	// the ORC default.
	CompressionBlockSize uint64
	// UseRLEv2 selects RLEv2 for integer, timestamp, decimal, and length streams.
	UseRLEv2 bool
	// UseDictionary writes STRING, CHAR, and VARCHAR values using a per-stripe dictionary.
	UseDictionary bool
	// MaxBufferedBytes sets the approximate value-buffer limit for a stripe.
	// Zero uses 16 MiB. A single row can exceed the limit.
	MaxBufferedBytes uint64
	// MaxRowsPerStripe also flushes a stripe after this many rows. Zero imposes
	// no row-count limit; MaxBufferedBytes still applies.
	MaxRowsPerStripe uint64
	// RowIndexStride sets the maximum rows per row-index entry. Zero uses the
	// ORC default of 10,000 rows. Values below 1,000 are not supported.
	RowIndexStride uint64
	// DisableRowIndexes omits row-index streams and sets the stride to zero.
	DisableRowIndexes bool
	// BloomFilterColumns enables row-group bloom filters for the named root
	// columns. Supported types are integer, floating-point, string, and binary.
	BloomFilterColumns []string
	// BloomFilterFPP sets the requested false-positive probability. Zero uses
	// the ORC default of 0.01.
	BloomFilterFPP float64
	// LegacyBloomFilterEncoding writes the original fixed64 bitset format for
	// older ORC readers. The default uses the UTF-8 byte bitset format.
	LegacyBloomFilterEncoding bool
	// EncryptColumns encrypts each named root field and its type subtree.
	EncryptColumns []EncryptionRule
	// KeyProvider supplies Hadoop master keys named by EncryptColumns.
	KeyProvider KeyProvider
	// EncryptionKeyProvider wraps local keys using a provider-specific format,
	// such as an external KMS integration. Set this instead of KeyProvider.
	EncryptionKeyProvider EncryptionKeyProvider
	// SoftwareVersion adds a descriptive writer version to the ORC footer.
	// Empty omits the field.
	SoftwareVersion string
}

WriterOptions controls file compression and column encodings. UseRLEv2 selects RLEv2 streams. UseDictionary selects dictionary encoding for STRING, CHAR, and VARCHAR columns.

Source Files

View all Source files

Directories

Path Synopsis
internal
rlev1
Package rlev1 implements the integer RLEv1 stream encoding used by ORC.
 Package rlev1 implements the integer RLEv1 stream encoding used by ORC.
rlev2
Package rlev2 implements the integer RLEv2 stream encoding used by ORC.
 Package rlev2 implements the integer RLEv2 stream encoding used by ORC.
wire
Package wire reads the small subset of Protocol Buffers wire format used by ORC metadata.
 Package wire reads the small subset of Protocol Buffers wire format used by ORC metadata.
interop
write command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.