orc

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 5, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

README

orc-go

Read and write Apache ORC files in Go. Reader decodes ORC data into Go values; Writer encodes Go values as ORC. The package supports nested types, compression, indexes, bloom filters, metadata, and encrypted columns.

Most ORC files store each row as a STRUCT. In Go, use orc.Row to access fields by name. For files with another root type, use the value-based methods.

The project is under development, so its API may change. It requires Go 1.22 or newer.

Contents

Install

go get github.com/fallintoplace/orc-go

Quick start

This example writes one row to people.orc, then reads it back. The schema describes a row with an id and a name:

STRUCT
├── id: LONG
└── name: STRING

In the schema list below, type 0 is the root. Its Subtypes point to the LONG and STRING entries, and FieldNames gives those fields their names.

package main

import (
	"fmt"
	"log"
	"os"

	orc "github.com/fallintoplace/orc-go"
)

func main() {
	file, err := os.Create("people.orc")
	if err != nil {
		log.Fatal(err)
	}
	defer file.Close()

	schema := []orc.Type{
		{Kind: orc.TypeKindStruct, Subtypes: []uint32{1, 2}, FieldNames: []string{"id", "name"}},
		{Kind: orc.TypeKindLong},
		{Kind: orc.TypeKindString},
	}

	writer, err := orc.NewWriter(file, schema, orc.WriterOptions{})
	if err != nil {
		log.Fatal(err)
	}
	if err := writer.WriteRow(orc.Row{"id": int64(1), "name": "Ada"}); err != nil {
		log.Fatal(err)
	}
	if err := writer.Close(); err != nil {
		log.Fatal(err)
	}

	reader, err := orc.Open("people.orc")
	if err != nil {
		log.Fatal(err)
	}
	defer reader.Close()

	rows, err := reader.ReadAll()
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(rows[0]["name"]) // Ada
}

Use WriteRow and the row-reading methods when the root type is STRUCT. For another root type, such as a scalar, list, or map, use WriteValue and the value-reading methods. Missing fields and nil values become ORC nulls. writer.Close() finishes the file; it does not close the destination.

Read data

Use ReadRows to process rows one at a time. It keeps one stripe in memory. Use ReadAll when the full result fits in memory.

err := reader.ReadRows(func(row orc.Row) error {
	fmt.Println(row["name"])
	return nil
})

To read only some rows or fields:

  • ReadRowsRange and ReadAllRange read a row range. Row numbers start at zero; the start is included and the end is excluded. For example, [10, 20) returns rows 10 through 19.
  • ReadRowsWithColumns and ReadAllWithColumns read selected top-level fields. A selected nested field is returned in full.
  • Range reads also have WithColumns versions, so you can combine a row range with selected fields.

For a file whose root is not a STRUCT, use ReadValues to stream values or ReadAllValues to load them into memory. Use ReadValuesRange or ReadAllValuesRange to read a range. A STRUCT value is returned as orc.Row; other values use the Go types below.

Go values

The reader returns these Go types. An ORC null is returned as nil.

ORC type Go value
BOOLEAN bool
BYTE, SHORT, INT, LONG int8, int16, int32, int64
FLOAT, DOUBLE float32, float64
STRING, CHAR, VARCHAR string
BINARY []byte
GEOMETRY, GEOGRAPHY []byte (WKB)
DATE, TIMESTAMP, TIMESTAMP_INSTANT time.Time
DECIMAL orc.Decimal
STRUCT orc.Row (map[string]any)
LIST []any
MAP []orc.MapEntry
UNION orc.UnionValue

Nested values use the same Go types:

row := orc.Row{
	"profile": orc.Row{"name": "Ada", "scores": []any{int32(3), int32(8)}},
	"labels":  []orc.MapEntry{{Key: "city", Value: "Paris"}},
	"choice":  orc.UnionValue{Tag: 0, Value: "text alternative"},
}

Maps are slices of key/value pairs. This keeps their entry order and supports keys that Go maps cannot use. UnionValue.Tag is the zero-based index of the selected alternative in the schema.

Type-specific rules:

  • GEOMETRY and GEOGRAPHY: Values are WKB byte slices. Set Type.CRS to choose a coordinate reference system; an empty value means OGC:CRS84. GEOGRAPHY uses spherical edges by default. Set Type.EdgeInterpolation to choose another supported method.
  • DECIMAL: Use orc.Decimal or parse a string with orc.ParseDecimal. Set Precision and Scale in the schema to require a fixed scale. The writer rejects values that would need rounding. If both are zero, values keep their own scale and have no precision limit.
  • DATE and timestamps: DATE uses the date in the input time.Time and returns it in UTC. TIMESTAMP keeps the calendar and clock fields using the stripe's writer timezone. TIMESTAMP_INSTANT keeps the instant.
  • CHAR and VARCHAR: CHAR is truncated and padded when written; trailing spaces are removed when read. VARCHAR is truncated when written. Both lengths count Unicode code points.

The writer uses Gregorian calendar rules. Older ORC files may name a different calendar; the reader follows it. If the file does not name a calendar, the reader assumes Julian/Gregorian. Set ReaderOptions.DefaultCalendar to choose a different fallback. Dates that exist only in the Julian calendar, such as February 29, 1500, become the next valid Gregorian date.

Compression and writer options

Compression and encoding

Both reader and writer support NONE, ZLIB, SNAPPY, LZO, LZ4, ZSTD, and BROTLI compression. The writer uses NONE by default.

The reader accepts RLEv1 and RLEv2 for integer, date, timestamp, and decimal streams. The writer uses RLEv1 by default; set UseRLEv2 to write RLEv2. For STRING, CHAR, and VARCHAR, the reader accepts direct and dictionary encodings. The writer uses direct encoding by default; set UseDictionary to use a dictionary in each stripe. BINARY columns use direct encoding.

Stripe, index, and filter settings

ORC stores rows in groups called stripes. Use these options to control how the writer creates them:

  • MaxRowsPerStripe starts a new stripe after the chosen number of rows. Zero writes all rows in one stripe when Close is called.
  • RowIndexStride sets the maximum rows covered by each row-index entry. The default is 10,000; values below 1,000 are not supported.
  • DisableRowIndexes omits row indexes. Bloom filters need row indexes.
  • BloomFilterColumns adds filters to selected top-level integer, floating-point, string, or binary fields. A filter can report a possible match that is not present, so check the data before relying on it. The false-positive probability is 0.01 by default; set BloomFilterFPP to change it.
  • LegacyBloomFilterEncoding writes the older fixed64 bitset format.
  • CompressionBlockSize sets the maximum uncompressed chunk size. Zero uses the ORC default.
  • EncryptColumns encrypts selected top-level fields and their nested fields. See Encrypt columns.
  • SoftwareVersion adds a descriptive version string to the file footer.

Inspect a file

Use Reader to inspect a file without reading all its values:

  • Schema() returns the ORC schema; Stripes() returns stripe locations and row counts.
  • FileStatistics() and StripeStatistics() return file and stripe statistics.
  • RowIndexes() and BloomFilterIndexes() return indexes when they are present.
  • UserMetadata() returns application-defined metadata. Use AddUserMetadata(name, value) when writing.
  • WriterMetadata() returns the writer ID and version details. EncryptionMasks() returns the fallback masks stored in the file.

String statistics may contain exact minimum and maximum values or bounds for long strings. LIST and MAP statistics include child counts. Geospatial statistics include bounding boxes and WKB type codes when the file stores them.

Encrypt columns

Set EncryptColumns to encrypt a top-level field and all its nested fields. The reader and writer need access to the same key. For Hadoop key wrapping, the master key must be 16 bytes for AES-CTR 128 or 32 bytes for AES-CTR 256.

Use KeyProvider for Hadoop key wrapping. To connect AWS, GCP, Azure, or another KMS, implement EncryptionKeyProvider; this package does not include cloud SDKs. The file stores wrapped local keys, not the master key.

Here is a Hadoop key provider example. It assumes file is open and schema has a top-level email field. This key is for demonstration only; load production keys from secure storage.

masterKey := []byte("0123456789abcdef") // 16 bytes for AES-CTR 128
keys := orc.KeyProviderFunc(func(name string, version uint32) ([]byte, error) {
	if name != "customer-data" || version != 1 {
		return nil, orc.ErrKeyUnavailable
	}
	return masterKey, nil
})

writer, err := orc.NewWriter(file, schema, orc.WriterOptions{
	EncryptColumns: []orc.EncryptionRule{{
		Column: "email", KeyName: "customer-data", KeyVersion: 1,
		Algorithm: orc.EncryptionAESCTR128,
	}},
	KeyProvider: keys,
})

Pass the same provider when opening the file:

reader, err := orc.OpenWithOptions(path, orc.ReaderOptions{KeyProvider: keys})

Without a key, the reader returns fallback data for encrypted columns. This writer uses nulls as fallback data, so file and stripe statistics reflect those nulls. With a key, EncryptionStatistics("email") returns statistics for the encrypted field and its nested fields. The reader also uses encrypted row indexes and bloom filters when present. User metadata is not encrypted.

AES-CTR does not detect tampering. Verify file integrity separately if you need tamper detection.

License

Apache License 2.0. See LICENSE.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrInvalidFile is returned when an ORC file's metadata is malformed.
	ErrInvalidFile = errors.New("invalid ORC file")
	// ErrUnsupportedCompression is returned for an ORC compression codec that is not supported.
	ErrUnsupportedCompression = errors.New("unsupported ORC compression")
	// ErrUnsupportedCalendar is returned for an ORC calendar kind that is not supported.
	ErrUnsupportedCalendar = errors.New("unsupported ORC calendar")
)
View Source 
var (
	// ErrUnsupportedType is returned when row reading encounters a type that is
	// not implemented yet.
	ErrUnsupportedType = errors.New("unsupported ORC type")
	// ErrUnsupportedEncoding is returned when row reading encounters a column
	// encoding that is not implemented yet.
	ErrUnsupportedEncoding = errors.New("unsupported ORC column encoding")
)
View Source 
var ErrColumnNotEncrypted = errors.New("ORC column is not encrypted")

ErrColumnNotEncrypted is returned when a column has no encrypted variant.

View Source 
var ErrEncryptedStatisticsUnavailable = errors.New("encrypted ORC statistics are unavailable")

ErrEncryptedStatisticsUnavailable is returned when encrypted statistics are not stored for a column or are incomplete.

View Source 
var ErrKeyUnavailable = errors.New("ORC encryption key is unavailable")

ErrKeyUnavailable tells a reader that a key is not available and the encrypted column's masked values should be used.

View Source 
var ErrUnsupportedEncryption = errors.New("unsupported ORC encryption")

ErrUnsupportedEncryption is returned for encryption metadata or algorithms that this implementation cannot read.

View Source 
var ErrWriterClosed = errors.New("ORC writer is closed")

ErrWriterClosed is returned when a writer operation is called after Close.

Functions

This section is empty.

Types

type BinaryStatistics 

type BinaryStatistics struct {
	Sum    int64
	HasSum bool
}

BinaryStatistics describes binary columns.

type BloomFilter 

type BloomFilter struct {
	NumHashFunctions uint32
	Bitset           []uint64
	UTF8Bitset       []byte
}

BloomFilter is one row group's ORC bloom filter. Bitset contains the original repeated-fixed64 encoding; UTF8Bitset contains the byte encoding introduced by ORC-101. A filter can contain either or both representations.

func (BloomFilter) MightContainBytes 

func (f BloomFilter) MightContainBytes(value []byte) bool

MightContainBytes reports whether a byte sequence may be present in this filter. A true result can be a false positive; a false result is definitive.

func (BloomFilter) MightContainFloat64 

func (f BloomFilter) MightContainFloat64(value float64) bool

MightContainFloat64 reports whether a floating-point value may be present in this filter.

func (BloomFilter) MightContainInt64 

func (f BloomFilter) MightContainInt64(value int64) bool

MightContainInt64 reports whether an integer may be present in this filter.

func (BloomFilter) MightContainString 

func (f BloomFilter) MightContainString(value string) bool

MightContainString reports whether a string may be present in this filter. A true result can be a false positive; a false result is definitive.

type BucketStatistics 

type BucketStatistics struct {
	Counts []uint64
}

BucketStatistics contains the counts for boolean values, in false/true order.

type CalendarKind 

type CalendarKind uint32

CalendarKind identifies the calendar used to encode historical dates and timestamps in an ORC file. 

const (
	// CalendarKindUnknown means that the ORC file does not name a calendar.
	CalendarKindUnknown CalendarKind = iota
	// CalendarKindJulianGregorian uses the Julian calendar before October 1582
	// and the Gregorian calendar from October 15, 1582 onward.
	CalendarKindJulianGregorian
	// CalendarKindProlepticGregorian uses the Gregorian calendar for all dates.
	CalendarKindProlepticGregorian
)

type CollectionStatistics 

type CollectionStatistics struct {
	MinimumChildren    uint64
	HasMinimumChildren bool
	MaximumChildren    uint64
	HasMaximumChildren bool
	TotalChildren      uint64
	HasTotalChildren   bool
}

CollectionStatistics describes the number of children in LIST and MAP values. Has fields distinguish an omitted count from a valid zero count.

type ColumnBloomFilterIndex 

type ColumnBloomFilterIndex struct {
	ColumnID uint32
	// BloomEncoding is 0 for original filters and 1 for UTC timestamp filters.
	// Other values are preserved for forward compatibility.
	BloomEncoding uint32
	Filters       []BloomFilter
}

ColumnBloomFilterIndex contains row-group bloom filters for one ORC type id.

type ColumnEncryptionStatistics 

type ColumnEncryptionStatistics struct {
	File    []ColumnStatistics
	Stripes [][]ColumnStatistics
}

ColumnEncryptionStatistics contains unmasked statistics for an encrypted root field and its type subtree. File and each stripe's statistics are in schema type-id order, starting with the encrypted root field.

type ColumnRowIndex 

type ColumnRowIndex struct {
	ColumnID uint32
	Entries  []RowIndexEntry
}

ColumnRowIndex contains the row groups indexed for one ORC type id.

type ColumnStatistics 

type ColumnStatistics struct {
	// NumberOfValues is the number of non-null values recorded for this column.
	NumberOfValues    uint64
	HasNumberOfValues bool
	// HasNull reports whether the column includes any null values.
	HasNull bool
	// HasNullSet reports whether the file explicitly stores the hasNull field.
	HasNullSet bool
	// BytesOnDisk is the encoded size recorded for this column, when available.
	BytesOnDisk    uint64
	HasBytesOnDisk bool
	// The type-specific statistics below are nil when the file does not include
	// the corresponding statistic.
	Integer    *IntegerStatistics
	Double     *DoubleStatistics
	String     *StringStatistics
	Bucket     *BucketStatistics
	Decimal    *DecimalStatistics
	Date       *DateStatistics
	Binary     *BinaryStatistics
	Timestamp  *TimestampStatistics
	Collection *CollectionStatistics
	Geospatial *GeospatialStatistics
}

ColumnStatistics contains the statistics recorded for one ORC type id. Type-specific fields are nil when the file does not include that statistic.

type CompressionKind 

type CompressionKind uint32

CompressionKind identifies the codec used for ORC file sections and streams. 

const (
	CompressionNone CompressionKind = iota
	CompressionZlib
	CompressionSnappy
	CompressionLZO
	CompressionLZ4
	CompressionZstd
	CompressionBrotli
)

func (CompressionKind) String 

func (kind CompressionKind) String() string

type DateStatistics 

type DateStatistics struct {
	Minimum    int32
	HasMinimum bool
	Maximum    int32
	HasMaximum bool
}

DateStatistics stores the minimum and maximum day counts since 1970-01-01. The calendar used to interpret those counts is reported by Reader.CalendarKind.

type Decimal 

type Decimal string

Decimal is an exact base-10 value. Its string form uses an optional sign and decimal point, without an exponent (for example, Decimal("-12.340")). The named type keeps decimal values distinct from ORC STRING values.

func ParseDecimal 

func ParseDecimal(value string) (Decimal, error)

ParseDecimal validates a decimal string and returns it without changing its spelling or scale.

func (Decimal) String 

func (value Decimal) String() string

String returns the exact decimal text.

type DecimalStatistics 

type DecimalStatistics struct {
	Minimum    string
	HasMinimum bool
	Maximum    string
	HasMaximum bool
	Sum        string
	HasSum     bool
}

DecimalStatistics describes decimal columns using exact ORC decimal text.

type DoubleStatistics 

type DoubleStatistics struct {
	Minimum    float64
	HasMinimum bool
	Maximum    float64
	HasMaximum bool
	Sum        float64
	HasSum     bool
}

DoubleStatistics describes floating-point columns.

type EdgeInterpolationAlgorithm 

type EdgeInterpolationAlgorithm uint32

EdgeInterpolationAlgorithm selects how geography edges are interpreted. 

const (
	// EdgeInterpolationSpherical is the default spherical geodesic algorithm.
	EdgeInterpolationSpherical EdgeInterpolationAlgorithm = iota
	EdgeInterpolationVincenty
	EdgeInterpolationThomas
	EdgeInterpolationAndoyer
	EdgeInterpolationKarney
)

type EncryptionAlgorithm 

type EncryptionAlgorithm uint32

EncryptionAlgorithm selects the AES key size used for ORC column streams. 

const (
	// EncryptionAESCTR128 uses AES-CTR with a 128-bit local key.
	EncryptionAESCTR128 EncryptionAlgorithm = 1
	// EncryptionAESCTR256 uses AES-CTR with a 256-bit local key.
	EncryptionAESCTR256 EncryptionAlgorithm = 2
)

type EncryptionKeyProvider 

type EncryptionKeyProvider interface {
	// Kind identifies the key provider format written to the ORC file.
	Kind() KeyProviderKind
	// GenerateLocalKey creates an AES key and the opaque bytes needed to
	// recover it later.
	GenerateLocalKey(name string, version uint32, algorithm EncryptionAlgorithm) (localKey, encryptedLocalKey []byte, err error)
	// DecryptLocalKey recovers a local AES key from its opaque bytes.
	DecryptLocalKey(name string, version uint32, algorithm EncryptionAlgorithm, encryptedLocalKey []byte) ([]byte, error)
}

EncryptionKeyProvider wraps and unwraps ORC local encryption keys. Use this interface to connect a KMS without adding its SDK to this package. The returned local key must be 16 bytes for AES-CTR-128 or 32 bytes for AES-CTR-256. The encrypted local key is stored in the ORC file as opaque bytes. Kind must be a provider kind defined by ORC. Return ErrKeyUnavailable when a reader cannot access the key.

type EncryptionMask 

type EncryptionMask struct {
	// Name is the mask name.
	Name string
	// Parameters contains the mask's stored parameters in order.
	Parameters []string
	// Columns contains unencrypted root type IDs covered by the mask.
	Columns []uint32
}

EncryptionMask describes a fallback data mask recorded in ORC encryption metadata. Columns contains the unencrypted root column IDs for the mask.

type EncryptionRule 

type EncryptionRule struct {
	Column     string
	KeyName    string
	KeyVersion uint32
	Algorithm  EncryptionAlgorithm
}

EncryptionRule encrypts a root field and its complete type subtree. Column names are root STRUCT field names.

type GeospatialBoundingBox 

type GeospatialBoundingBox struct {
	XMin    float64
	HasXMin bool
	XMax    float64
	HasXMax bool
	YMin    float64
	HasYMin bool
	YMax    float64
	HasYMax bool
	ZMin    float64
	HasZMin bool
	ZMax    float64
	HasZMax bool
	MMin    float64
	HasMMin bool
	MMax    float64
	HasMMax bool
}

GeospatialBoundingBox contains the minimum and maximum coordinate for each axis present in the column's geospatial values.

type GeospatialStatistics 

type GeospatialStatistics struct {
	BoundingBox     *GeospatialBoundingBox
	GeospatialTypes []int32
}

GeospatialStatistics describes the values in a GEOMETRY or GEOGRAPHY column.

type IntegerStatistics 

type IntegerStatistics struct {
	Minimum    int64
	HasMinimum bool
	Maximum    int64
	HasMaximum bool
	Sum        int64
	HasSum     bool
}

IntegerStatistics describes integer, byte, and long columns.

type KeyProvider 

type KeyProvider interface {
	GetKey(name string, version uint32) ([]byte, error)
}

KeyProvider resolves a named and versioned AES master key for Hadoop ORC key wrapping. Use EncryptionKeyProvider for other wrapping formats. The ORC file stores wrapped local keys, never the master key itself.

type KeyProviderFunc 

type KeyProviderFunc func(name string, version uint32) ([]byte, error)

KeyProviderFunc adapts a function to the KeyProvider interface.

func (KeyProviderFunc) GetKey 

func (f KeyProviderFunc) GetKey(name string, version uint32) ([]byte, error)

GetKey resolves a key using f.

type KeyProviderKind 

type KeyProviderKind uint32

KeyProviderKind identifies how the encrypted local keys in an ORC file are wrapped. 

const (
	// KeyProviderKindUnknown is used when the file does not name a provider.
	KeyProviderKindUnknown KeyProviderKind = iota
	// KeyProviderKindHadoop uses Hadoop or Ranger KMS wrapping.
	KeyProviderKindHadoop
	// KeyProviderKindAWS uses AWS KMS wrapping.
	KeyProviderKindAWS
	// KeyProviderKindGCP uses Google Cloud KMS wrapping.
	KeyProviderKindGCP
	// KeyProviderKindAzure uses Azure Key Vault wrapping.
	KeyProviderKindAzure
)

type MapEntry 

type MapEntry struct {
	Key   any
	Value any
}

MapEntry is one key/value pair in an ORC MAP value. A slice preserves entry order and supports key types that cannot be Go map keys.

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads ORC file metadata and decodes root values from its stripes.

func NewReader 

func NewReader(source io.ReaderAt, size int64) (*Reader, error)

NewReader reads an ORC file from a random access source. The caller retains ownership of source and is responsible for closing it if needed.

func NewReaderWithOptions 

func NewReaderWithOptions(source io.ReaderAt, size int64, options ReaderOptions) (*Reader, error)

NewReaderWithOptions reads an ORC file from a random access source. The caller retains ownership of source and is responsible for closing it.

func Open 

func Open(path string) (*Reader, error)

Open opens an ORC file from disk.

func OpenWithOptions 

func OpenWithOptions(path string, options ReaderOptions) (*Reader, error)

OpenWithOptions opens a file with reader options.

func (*Reader) BloomFilterIndexes 

func (r *Reader) BloomFilterIndexes() ([]StripeBloomFilterIndexes, error)

BloomFilterIndexes reads BLOOM_FILTER and BLOOM_FILTER_UTF8 streams from each stripe. Encrypted filters are decoded when a key is available; without a key, encrypted columns expose only their unencrypted masked filters.

func (*Reader) CalendarKind 

func (r *Reader) CalendarKind() CalendarKind

CalendarKind reports the calendar used to interpret DATE and TIMESTAMP values. A known footer value takes priority; otherwise the reader uses the configured fallback or Julian/Gregorian by default.

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the file when the Reader was created with Open.

func (*Reader) Compression 

func (r *Reader) Compression() CompressionKind

Compression returns the codec declared by the file postscript.

func (*Reader) EncryptionMasks 

func (r *Reader) EncryptionMasks() []EncryptionMask

EncryptionMasks returns the fallback data masks declared in the file. The returned slices do not alias the reader's encryption metadata.

func (*Reader) EncryptionStatistics 

func (r *Reader) EncryptionStatistics(column string) (ColumnEncryptionStatistics, error)

EncryptionStatistics returns the unmasked file and stripe statistics for an encrypted root field. A key provider must be configured on the Reader.

func (*Reader) FileStatistics 

func (r *Reader) FileStatistics() []ColumnStatistics

FileStatistics returns the statistics for each column in type-id order. Values in encrypted columns use the unencrypted masked statistics, when present. Encrypted statistics are not exposed by this API.

func (*Reader) FormatVersion 

func (r *Reader) FormatVersion() []uint32

FormatVersion returns the ORC file version declared by the postscript.

func (*Reader) NumRows 

func (r *Reader) NumRows() uint64

NumRows returns the total number of rows declared by the file footer.

func (*Reader) NumStripes 

func (r *Reader) NumStripes() int

NumStripes returns the number of stripes in the file.

func (*Reader) ReadAll 

func (r *Reader) ReadAll() ([]Row, error)

ReadAll reads all rows into memory. Use ReadRows for large files.

func (*Reader) ReadAllRange 

func (r *Reader) ReadAllRange(start, end uint64) ([]Row, error)

ReadAllRange reads the rows in [start, end) into memory.

func (*Reader) ReadAllRangeWithColumns 

func (r *Reader) ReadAllRangeWithColumns(start, end uint64, names []string) ([]Row, error)

ReadAllRangeWithColumns reads [start, end) with only the named root fields.

func (*Reader) ReadAllValues 

func (r *Reader) ReadAllValues() ([]any, error)

ReadAllValues reads all values of the root type into memory. Use ReadValues for large files.

func (*Reader) ReadAllValuesRange 

func (r *Reader) ReadAllValuesRange(start, end uint64) ([]any, error)

ReadAllValuesRange reads root values from the zero-based range [start, end) into memory.

func (*Reader) ReadAllWithColumns 

func (r *Reader) ReadAllWithColumns(names []string) ([]Row, error)

ReadAllWithColumns reads all rows with only the named root fields. Use ReadRowsWithColumns for large files.

func (*Reader) ReadRows 

func (r *Reader) ReadRows(yield func(Row) error) error

ReadRows calls yield for each root STRUCT value in file order. A null root STRUCT is returned as a nil Row. It keeps decoded values for one stripe in memory at a time. Returning an error from yield stops the read.

func (*Reader) ReadRowsRange 

func (r *Reader) ReadRowsRange(start, end uint64, yield func(Row) error) error

ReadRowsRange calls yield for the rows in [start, end), using zero-based file row numbers. Stripes outside the range are not read.

func (*Reader) ReadRowsRangeWithColumns 

func (r *Reader) ReadRowsRangeWithColumns(start, end uint64, names []string, yield func(Row) error) error

ReadRowsRangeWithColumns reads [start, end) and calls yield with only the named root fields. Selecting a nested field's root returns its full value.

func (*Reader) ReadRowsWithColumns 

func (r *Reader) ReadRowsWithColumns(names []string, yield func(Row) error) error

ReadRowsWithColumns calls yield with only the named root fields. Selecting a nested field's root returns the full nested value for that field.

func (*Reader) ReadValues 

func (r *Reader) ReadValues(yield func(any) error) error

ReadValues calls yield for each value of the root type in file order. It keeps decoded values for one stripe in memory at a time. Root STRUCT values are returned as Row; other values use the Go type for their ORC type.

func (*Reader) ReadValuesRange 

func (r *Reader) ReadValuesRange(start, end uint64, yield func(any) error) error

ReadValuesRange reads root values from the zero-based range [start, end). Stripes outside the range are skipped.

func (*Reader) RowIndexStride 

func (r *Reader) RowIndexStride() uint64

RowIndexStride returns the row group size declared by the file footer.

func (*Reader) RowIndexes 

func (r *Reader) RowIndexes() ([]StripeRowIndexes, error)

RowIndexes reads and decodes the ROW_INDEX streams in each stripe. It does not load column data streams. Encrypted indexes are decoded when a key is available; otherwise only unencrypted masked indexes are returned.

func (*Reader) Schema 

func (r *Reader) Schema() []Type

Schema returns a copy of the file's ORC type tree.

func (*Reader) StripeStatistics 

func (r *Reader) StripeStatistics() [][]ColumnStatistics

StripeStatistics returns the statistics for each stripe, with columns in type-id order. The returned nested slices do not alias reader state.

func (*Reader) Stripes 

func (r *Reader) Stripes() []StripeInfo

Stripes returns a copy of the file's stripe directory.

func (*Reader) UserMetadata 

func (r *Reader) UserMetadata() []UserMetadataItem

UserMetadata returns the application-defined key/value pairs in the file footer. The returned values do not alias the reader's metadata.

func (*Reader) WriterMetadata 

func (r *Reader) WriterMetadata() WriterMetadata

WriterMetadata returns the writer ID, writer version, and software version declared in the file, preserving whether each optional field was present.

type ReaderOptions 

type ReaderOptions struct {
	// DefaultCalendar is used for DATE and TIMESTAMP values when the file has
	// no calendar metadata. Zero defaults to the Julian/Gregorian calendar.
	DefaultCalendar CalendarKind
	// KeyProvider resolves Hadoop master keys used to wrap ORC local keys. If
	// nil and no matching EncryptionKeyProvider is set, encrypted columns use
	// their unencrypted masked variant. When both are set for a Hadoop file,
	// KeyProvider is tried if EncryptionKeyProvider reports the key unavailable.
	KeyProvider KeyProvider
	// EncryptionKeyProvider handles the matching ORC key-provider format. It is
	// tried before KeyProvider and can connect a KMS without adding its SDK.
	EncryptionKeyProvider EncryptionKeyProvider
}

ReaderOptions controls optional reader behavior.

type Row 

type Row map[string]any

Row contains one ORC root-struct value. A nil Row represents a null root STRUCT. Nested STRUCT values also use Row, LIST values use slices, MAP values use []MapEntry, and UNION values use UnionValue. Primitive values use bool for BOOLEAN, the matching integer width for integer columns, time.Time for DATE and timestamps, Decimal for DECIMAL, float32 or float64 for FLOAT or DOUBLE, string for STRING, CHAR, or VARCHAR, and []byte for BINARY, GEOMETRY, or GEOGRAPHY. Geospatial values use ORC's WKB representation. Timestamp values use UTC; TIMESTAMP preserves wall-clock fields while TIMESTAMP_INSTANT preserves the instant. A nil value represents an ORC null.

type RowIndexEntry 

type RowIndexEntry struct {
	Positions     []uint64
	Statistics    ColumnStatistics
	HasStatistics bool
}

RowIndexEntry contains the stream positions and optional statistics for one row group in an ORC column.

type StringStatistics 

type StringStatistics struct {
	Minimum       string
	HasMinimum    bool
	Maximum       string
	HasMaximum    bool
	LowerBound    string
	HasLowerBound bool
	UpperBound    string
	HasUpperBound bool
	Sum           int64
	HasSum        bool
}

StringStatistics describes string-like columns. HasMinimum and HasMaximum are false when ORC stores only a truncated bound. Sum is the total UTF-8 byte length of the values, as stored by ORC.

type StripeBloomFilterIndexes 

type StripeBloomFilterIndexes struct {
	Columns []ColumnBloomFilterIndex
}

StripeBloomFilterIndexes contains the bloom filter indexes found in one stripe. Columns are ordered by type id; columns without a bloom-filter stream are omitted.

type StripeInfo 

type StripeInfo struct {
	Offset       uint64
	IndexLength  uint64
	DataLength   uint64
	FooterLength uint64
	NumberOfRows uint64
}

StripeInfo contains the byte ranges and row count for one stripe.

type StripeRowIndexes 

type StripeRowIndexes struct {
	Columns []ColumnRowIndex
}

StripeRowIndexes contains the row indexes found in one stripe. Columns are ordered by type id; columns without a ROW_INDEX stream are omitted.

type TimestampStatistics 

type TimestampStatistics struct {
	Minimum         int64
	HasMinimum      bool
	Maximum         int64
	HasMaximum      bool
	MinimumUTC      int64
	HasMinimumUTC   bool
	MaximumUTC      int64
	HasMaximumUTC   bool
	MinimumNanos    int32
	HasMinimumNanos bool
	MaximumNanos    int32
	HasMaximumNanos bool
}

TimestampStatistics stores timestamp minima and maxima as Unix milliseconds. The UTC fields are present in newer ORC writers and distinguish instant comparisons from wall-clock timestamp comparisons. The nanosecond fields store the final six digits below millisecond precision, as values from 0 to 999999. When a nanosecond field is absent, ORC uses 0 for a minimum and 999999 for a maximum.

type Type 

type Type struct {
	Kind       TypeKind
	Subtypes   []uint32
	FieldNames []string
	// Attributes contains ORC key/value annotations for this type.
	Attributes []TypeAttribute
	// MaxLength limits CHAR and VARCHAR values in Unicode code points.
	MaxLength uint32
	// Precision and Scale may both be zero for legacy variable-scale DECIMAL.
	Precision uint32
	Scale     uint32
	// CRS sets the coordinate reference system for GEOMETRY and GEOGRAPHY.
	// Empty uses the ORC default, OGC:CRS84.
	CRS string
	// EdgeInterpolation sets the GEOGRAPHY edge algorithm. Zero selects
	// EdgeInterpolationSpherical, the ORC default.
	EdgeInterpolation EdgeInterpolationAlgorithm
}

Type describes one node in the ORC type tree. Subtypes refer to indexes in the schema's Types slice; ORC stores the root type at index zero.

type TypeAttribute 

type TypeAttribute struct {
	Key   string
	Value string
}

TypeAttribute is a key/value annotation stored on an ORC type.

type TypeKind 

type TypeKind uint32

TypeKind identifies an ORC column type. 

const (
	TypeKindBoolean TypeKind = iota
	TypeKindByte
	TypeKindShort
	TypeKindInt
	TypeKindLong
	TypeKindFloat
	TypeKindDouble
	TypeKindString
	TypeKindBinary
	TypeKindTimestamp
	TypeKindList
	TypeKindMap
	TypeKindStruct
	TypeKindUnion
	TypeKindDecimal
	TypeKindDate
	TypeKindVarchar
	TypeKindChar
	TypeKindTimestampInstant
	TypeKindGeometry
	TypeKindGeography
)

func (TypeKind) String 

func (kind TypeKind) String() string

type UnionValue 

type UnionValue struct {
	Tag   uint8
	Value any
}

UnionValue stores the selected ORC UNION alternative. Tag is the zero-based index into the union type's Subtypes slice.

type UserMetadataItem 

type UserMetadataItem struct {
	Name  string
	Value []byte
}

UserMetadataItem is an application-defined key and byte value stored in an ORC file footer. Column encryption does not encrypt user metadata.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes ORC files for primitive or nested root values. It buffers values for the current stripe and writes completed stripes to the destination. Close writes the file footer and does not close the destination.

func NewWriter 

func NewWriter(destination io.Writer, schema []Type, options WriterOptions) (*Writer, error)

NewWriter creates a writer for a supported ORC type tree. The root may be any supported scalar or compound type. Use WriteRow for root STRUCT schemas and WriteValue for any root type.

func (*Writer) AddUserMetadata 

func (w *Writer) AddUserMetadata(name string, value []byte) error

AddUserMetadata appends an application-defined key/value pair to the file footer. Both the name and value are copied before the call returns. Column encryption does not encrypt user metadata.

func (*Writer) Close 

func (w *Writer) Close() error

Close writes the final stripe and ORC file footer. Calling Close again returns the result of the first call.

func (*Writer) WriteRow 

func (w *Writer) WriteRow(row Row) error

WriteRow adds one non-null root STRUCT row. Missing fields and nil values are written as ORC nulls. Use WriteValue to write a null root STRUCT. Binary and geospatial byte values are copied before they are buffered.

func (*Writer) WriteValue 

func (w *Writer) WriteValue(value any) error

WriteValue adds one value of the schema's root type. Use this method when the root is not a STRUCT, or when a root STRUCT itself may be null. For a root STRUCT, non-null values may be Row or map[string]any. Binary and geospatial byte values are copied before they are buffered.

type WriterMetadata 

type WriterMetadata struct {
	// ImplementationID is the registered ORC writer ID, when present.
	ImplementationID uint32
	// HasImplementationID reports whether the footer includes a writer ID.
	HasImplementationID bool
	// SoftwareVersion is the writer's descriptive version string, when present.
	SoftwareVersion string
	// HasSoftwareVersion reports whether the footer includes a software version.
	HasSoftwareVersion bool
	// WriterVersion is the writer's version sequence from the postscript.
	WriterVersion uint32
	// HasWriterVersion reports whether the postscript includes a writer version.
	HasWriterVersion bool
}

WriterMetadata identifies the writer declared in an ORC file footer and postscript.

type WriterOptions 

type WriterOptions struct {
	// Compression selects NONE, ZLIB, SNAPPY, LZO, LZ4, ZSTD, or BROTLI.
	Compression CompressionKind
	// CompressionBlockSize sets the maximum uncompressed chunk size. Zero uses
	// the ORC default.
	CompressionBlockSize uint64
	// UseRLEv2 selects RLEv2 for integer, timestamp, decimal, and length streams.
	UseRLEv2 bool
	// UseDictionary writes STRING, CHAR, and VARCHAR values using a per-stripe dictionary.
	UseDictionary bool
	// MaxRowsPerStripe flushes a stripe after this many rows. Zero writes all rows
	// as one stripe when Close is called.
	MaxRowsPerStripe uint64
	// RowIndexStride sets the maximum rows per row-index entry. Zero uses the
	// ORC default of 10,000 rows. Values below 1,000 are not supported.
	RowIndexStride uint64
	// DisableRowIndexes omits row-index streams and sets the stride to zero.
	DisableRowIndexes bool
	// BloomFilterColumns enables row-group bloom filters for the named root
	// columns. Supported types are integer, floating-point, string, and binary.
	BloomFilterColumns []string
	// BloomFilterFPP sets the requested false-positive probability. Zero uses
	// the ORC default of 0.01.
	BloomFilterFPP float64
	// LegacyBloomFilterEncoding writes the original fixed64 bitset format for
	// older ORC readers. The default uses the UTF-8 byte bitset format.
	LegacyBloomFilterEncoding bool
	// EncryptColumns encrypts each named root field and its type subtree.
	EncryptColumns []EncryptionRule
	// KeyProvider supplies Hadoop master keys named by EncryptColumns.
	KeyProvider KeyProvider
	// EncryptionKeyProvider wraps local keys using a provider-specific format,
	// such as an external KMS integration. Set this instead of KeyProvider.
	EncryptionKeyProvider EncryptionKeyProvider
	// SoftwareVersion adds a descriptive writer version to the ORC footer.
	// Empty omits the field.
	SoftwareVersion string
}

WriterOptions controls file compression and column encodings. UseRLEv2 selects RLEv2 streams. UseDictionary selects dictionary encoding for STRING, CHAR, and VARCHAR columns.

Source Files

View all Source files

Directories

Path Synopsis
internal
rlev1
Package rlev1 implements the integer RLEv1 stream encoding used by ORC.
 Package rlev1 implements the integer RLEv1 stream encoding used by ORC.
rlev2
Package rlev2 implements the integer RLEv2 stream encoding used by ORC.
 Package rlev2 implements the integer RLEv2 stream encoding used by ORC.
wire
Package wire reads the small subset of Protocol Buffers wire format used by ORC metadata.
 Package wire reads the small subset of Protocol Buffers wire format used by ORC metadata.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.