parquet

package
v0.0.40 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 31, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package parquet reads and writes Apache Parquet files.

A parquet file is columns of compressed pages with an index at the end saying where every one of them is. That index is the footer, and it is what makes the format worth reading: a query that wants three columns out of two hundred reads the footer, works out which pages hold those three, and never touches the rest of the file. Metadata is that footer, and ReadMetadata is how to get it.

The footer is a Thrift structure. This package reads that itself rather than pulling in a generated Thrift reader, for the same reason the Arrow IPC metadata is read by hand next door: a footer is somebody else's bytes, every length in it is a claim rather than a promise, and a reader for this job needs to bounds check every read and refuse the claims it cannot satisfy.

Metadata is enough to say what a file holds, how many rows it has, where each column chunk lives and what the writer said about its values, which is what a scan needs before it reads anything. Metadata.Schema turns the file's own schema into kuma types, and Metadata.Columns is the leaves of it with the levels a page decoder will need to put nulls and list boundaries back.

ReadPages goes one level down, walking the pages of a column chunk and handing back each header with the bytes behind it. The bytes are the ones in the file, compressed and encoded as the writer left them, so nothing here turns a page into values yet.

The decoders are what turns a page into values. RLEDecoder and BitPackedDecoder read the runs of small integers that nulls, list boundaries and dictionary indices are written as, which is the hybrid of repeated and packed runs parquet uses now and the plain packing it used before that. PlainDecoder reads the values themselves, written as they are, which is what every other encoding in the format is a way of not doing and what every one of them ends up at. DeltaDecoder reads the one that a column of integers is written in when a dictionary is not worth keeping, which is the differences between the values rather than the values. DeltaLengthDecoder and DeltaByteArrayDecoder are the same idea for a column of byte arrays: the first writes their lengths as differences and puts all the bytes behind them, and the second writes how much of each value the one in front of it already said, which is what turns a sorted column of keys into a few bytes a row.

PlainEncoder is the same encoding the other way round and is where writing a file starts. It writes the values of a page into a buffer, one method per physical type and no conversions, because a value written at the wrong width is not a wrong value but a different value at every position after it. There is nothing in it that writes an int96, since no writer has produced one for years and a timestamp in twelve bytes with no zone and no unit is a mistake the format has replaced twice over.

RLEEncoder is the levels and the dictionary indices written back out, and picking the runs is the whole of what it does: the same values are a legal file written as one run or as fifty, so what makes an encoder of this worth anything is that it writes the small one. A value that repeats eight times or more becomes a repeat and everything else is packed in groups of eight, which is what makes the levels of a column with no nulls three bytes however many rows it has. There is nothing that writes the encoding this one replaced, since that would be writing for a reader that stopped existing years ago.

WritePage is the walk in ReadPages turned around, and it is the one place in the writer where a mistake does not look like a mistake. A page has no length in front of its header, so a reader finds the second page by reading the first header and adding up, which means a compressed size that is one byte out does not produce a page that is slightly wrong but a chunk where every page after it is nonsense that a reader has no way to tell from a file that was never parquet. So the header is checked against the body it was handed rather than taken, by the same rule a header read out of a file is checked by, and a page this writes is a page this package would accept. The checksum is the one field a caller does not fill in, since a caller that computes its own is a caller that can get it wrong.

WriteMetadata is the footer, which is what turns encoded pages into a file. It writes the Thrift structure ReadMetadata reads, then how long it is, then the magic, which is the last of every parquet file. The field numbers are the whole of what makes that work: a footer written with the right numbers is read by anything and one written with the wrong numbers is read by nothing, so the reading and the writing of every structure are kept next to each other and every footer in testdata is round tripped through both. What a writer decides and a reader never does is which fields to leave out, since nearly everything in the format is optional and a field that is absent reads back as the absent value of its type.

Metadata.SetSchema is Metadata.Schema the other way round, and it is the first thing a writer does, because the leaves of a schema in the order they come out of it are the order every row group has to hold its chunks in and a file whose schema and chunks disagree reads as somebody else's columns rather than as a broken file. A group in parquet is not a node with children but a node followed by them, so writing a tree is appending the nodes in the order a reader takes them off. What kuma can say and the format cannot is refused rather than approximated, since a duration written as a plain int64 is a column that reads back as something else and a file that quietly lost a type is worse than one that was never written. What kuma has two of and parquet has one of is written as the one and comes back as it, which is what happens to large strings, large lists, fixed size lists and dictionaries.

Write is all of that in one call, which is what writing a file amounts to: a table goes in and a parquet file comes out. It goes down in one pass with nothing seeked back to, so the writer can be a pipe or a socket, and it holds one page rather than the file, so a table of a hundred million rows costs a page. The bookkeeping is the whole of the job, because every offset in the footer has to be where the thing it points at actually started and there is no length in front of a page for a reader to check it against. So there is one counter of bytes written and every offset in the footer is a copy of it, rather than the same number arrived at a second way by adding up sizes. WriteFile is the same over a named file.

What comes out is uncompressed and has no page index in it, and every chunk of a column with few enough distinct values is written as a dictionary page and indices into it. That is where most of the size of a parquet file goes: a chunk of ten million country codes holds two hundred and fifty strings once and ten million small integers pointing at them. The decision is taken per chunk and taken before anything is written, since a chunk that changed its mind part way is one the reader here refuses, and a boolean and a float never get one. WriteOptions.Plain writes every value as it is instead, which is the largest file anything will open and the floor the rest is built on.

What it does write is the statistics, and they are what turns a file into one a scan can skip most of. Every column chunk goes down with the smallest and largest value in it and a count of how many of its values are missing, and the footer says that every column compares the way the format defines for its type, since a file that leaves that out is one whose bounds a reader is right to ignore on half the types in it. The bounds are values out of the chunk rather than truncations of them, and a NaN is left out of them, a chunk bounded by one being a chunk that nothing can be filtered out of. So a file this writes is one the row group skipping below works on, and a filter for a value that lives in one row group of fifty reads the footer and one group.

ColumnReader is where the two halves meet. A page keeps its levels and its values apart and only the rows that have a value are written down, so putting a column back together means walking the two together and dropping the values in around the nulls. ReadColumn does that for a whole column chunk and hands back an array. A chunk that was written as indices into a dictionary, which is most of a real file, comes back dictionary encoded rather than expanded, since that is the shape it was written in and the shape the kernels would rather have it in. What it reads so far is a flat column of plain, dictionary or delta encoded pages, and anything else is refused by name rather than guessed at.

FileReader is the whole of it in one place. It reads the footer, holds it, and hands back the columns of one row group at a time. Which columns is what Project says, and a projection is the reason the format exists: a file of two hundred columns keeps each of them apart from the others, so a reader that wants three of them reads three runs of pages and never touches the rest. BytesRead is how a caller checks that, since a projection that quietly read the whole file would give the same answers at ten times the cost.

Bounds is the other half of not reading a file. A writer usually writes the smallest and largest value of every column chunk into the footer, so a scan carrying a filter can ask a row group what its columns hold and skip the whole group without opening a page of it. What makes that worth care is that parquet spent years without saying how its values compare, so a file holds two pairs of bounds written by two different rules and only one of them is worth reading on most types. ReadBounds is that rule, and FileReader.Bounds is it applied to a row group.

The page index is the same idea one level down. A writer that was asked for one writes the bounds of every page and the whereabouts of every page into two structures at the end of the file, so a scan that cannot skip a row group can still tell which of its pages a filter would keep. They are read a column at a time, because they are not in the footer and a scan filtering on one column of two hundred has no reason to read the index of the rest. ReadPageBounds is the two of them together and FileReader.PageBounds is it applied to a row group.

BloomFilter answers for the values bounds cannot. A range says nothing useful about a column of identifiers scattered across a file, since every row group covers a range with the wanted value somewhere in the middle of it, and yet only one group holds it. A writer that was asked for a filter hashed every value of a chunk into a bitset at the end of the file, and a reader hashing the value it wants looks at the same bits: a bit that is clear means the chunk never held it. The other answer is a maybe rather than a yes, so a filter is something to skip on and never to answer on. ReadBloomFilter is one chunk's and FileReader.BloomFilter is it applied to a row group.

Predicate is what turns all of that into an answer. It is one column compared against one value, which is what most of a filter on a scan is made of, and FileReader.RowGroups takes a list of them and gives back the row groups that may hold a matching row. Every group it leaves out is one whose statistics say it holds none, worked out from the footer and, where the writer wrote one, from a bloom filter. A group it returns may hold a matching row rather than does, and a file whose writer wrote no statistics gives back every group, so what comes out of it is where to look rather than what is there.

Options.Filter is both halves together and is what a caller reading a file wants. It skips the row groups the statistics rule out and compares the rows of the ones it reads, so what comes back is the rows that pass. The columns a predicate names do not have to be columns the caller asked for, since filtering on a timestamp nobody wants in the result is the ordinary case, and a file with no statistics gives the same rows for the cost it always took. Nothing there can change an answer, only how much of the file it took.

Decompressor is what undoes the compression of a page on the way through. Nearly every parquet file in the world is compressed and the codec is a property of a column chunk rather than of the file, so one file may hold a snappy column next to a gzip one next to one that was left alone. Those three are what is undone so far. Snappy is read by kuma/compress/snappy and gzip by the standard library, and a chunk written with a codec that is neither is refused by name.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrFormat is a file that is not a parquet file, or is one that has been
	// truncated or corrupted. It covers the magic at both ends, the footer
	// length, and everything the Thrift decoder refuses.
	ErrFormat = errors.New("bad parquet file")

	// ErrUnsupported is a file this package understands and cannot read yet.
	// An encrypted footer is the one that turns up in the wild.
	ErrUnsupported = errors.New("not supported yet")
)

The errors this package returns. Use errors.Is rather than comparing, since every one of them arrives wrapped in what was being read at the time.

Functions

func Read added in v0.0.15 

func Read(r io.ReaderAt, size int64, opts *Options) (*array.Table, error)

Read reads a whole parquet file.

The size is the size of the file, the same one ReadMetadata takes and for the same reason: a footer at the end of a file cannot be found by anything that only reads forwards. 

t, err := parquet.Read(r, size, &parquet.Options{Columns: []string{"id", "price"}})

Each column comes back in as many chunks as there were row groups holding rows, which is the chunking the file was written with and the chunking every kernel here is happy to read. A row group of no rows contributes nothing, and neither does one whose rows Options.Filter all rejected, so a filtered read of a file of a hundred row groups comes back in as many chunks as held a matching row.

func ReadColumn added in v0.0.9 

func ReadColumn(r io.ReaderAt, size int64, chunk *ColumnChunk, c Column) (*array.Array, error)

ReadColumn reads one column chunk of a file into an array.

The size is the size of the file, the same one ReadMetadata was given, and c is the column the chunk holds, which is one of the leaves Metadata.Columns returned. A chunk written as indices into a dictionary comes back dictionary encoded, which is what ColumnReader.Finish says more about.

This is one chunk read with a reader of its own. Something walking the row groups of a file reads the same column again and again and wants one reader for all of them, which is what ColumnReader.Chunk is.

func ReadFile added in v0.0.15 

func ReadFile(path string, opts *Options) (*array.Table, error)

ReadFile reads the file at path. It is Read over an open file, with the name of the file in any error the read returns.

Example

Reading a whole file. 

package main

import (
	"fmt"
	"log"

	"github.com/tamnd/kuma/parquet"
)

func main() {
	t, err := parquet.ReadFile("testdata/chunks.parquet", nil)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(t.NumRows(), "rows and", t.NumCols(), "columns")
	for i, f := range t.Schema.Fields {
		fmt.Printf("%s is a %s column in %d chunks\n", f.Name, f.Type, t.Columns[i].NumChunks())
	}
}

Output:
6 rows and 2 columns
code is a string column in 2 chunks
n is a int64 column in 2 chunks
Example (Dictionary)

Keeping the encoding of a column the file wrote as indices into a dictionary, which is worth doing for a column that repeats and not for one that does not. 

package main

import (
	"fmt"
	"log"

	"github.com/tamnd/kuma/parquet"
)

func main() {
	t, err := parquet.ReadFile("testdata/chunks.parquet", &parquet.Options{
		Columns:    []string{"code"},
		Dictionary: true,
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(t.Columns[0].DType())
}

Output:
dictionary<int32, string>
Example (Filter)

Reading the rows that pass a filter, which skips the row groups the footer rules out and compares the rows of the ones it reads. 

// The file holds twelve rows in three row groups with n running from nought
// to eleven, so only the last group can hold a row of eight or more and the
// other two are never opened. The filter names a column the caller did not
// ask for, which is the ordinary case, so n is read to compare the rows
// against and left out of the table.
t, err := parquet.ReadFile("testdata/stats.parquet", &parquet.Options{
	Columns: []string{"word"},
	Filter: []parquet.Predicate{
		parquet.Where("n", kernel.OpGe, int64(8)),
	},
})
if err != nil {
	log.Fatal(err)
}

fmt.Println(t.NumRows(), "rows of", t.Schema)
fmt.Println(text(t.Columns[0]))

Output:
4 rows of schema<word: string not null>
[zulu yankee victor sierra]
Example (Projection)

Reading two columns of a wide file and leaving the rest of it on disk. 

package main

import (
	"fmt"
	"log"

	"github.com/tamnd/kuma/parquet"
)

func main() {
	t, err := parquet.ReadFile("testdata/alltypes.parquet", &parquet.Options{
		Columns: []string{"name", "total"},
	})
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(t.Schema)
}

Output:
schema<name: string, total: int64>

func Write added in v0.0.20 

func Write(w io.Writer, t *array.Table, opts *WriteOptions) (int64, error)

Write writes a table as a parquet file and returns how many bytes it wrote. 

n, err := parquet.Write(w, t, nil)

The file is written forwards in one pass: the magic, then the pages of each row group a column at a time, then the footer that says where they all are. Nothing is seeked back to, so the writer can be a pipe or a network connection, and nothing is buffered beyond the page being built, so a table of a hundred million rows costs a page rather than a file.

Nothing is compressed. Each chunk of a column with few enough distinct values is written as a dictionary page and indices into it, which is where most of the size of a parquet file goes, and Options.Plain turns that off. A column whose type kuma has and parquet has not is refused by name rather than approximated, and so is a column that is a list, a map or a struct, since those need repetition levels that nothing here reads back yet.

A column the caller kept dictionary encoded is written as its values and gets whatever dictionary this writer decides on, because in parquet a dictionary is a decision about a chunk rather than a type. It comes back as its value type where the chunk was written plainly and as a dictionary of it where the chunk was not, which is the same rule every other file is read by.

func WriteFile added in v0.0.20 

func WriteFile(path string, t *array.Table, opts *WriteOptions) error

WriteFile writes a table to the file at path, creating it or truncating what is there. It is Write over a new file, with the name of the file in any error it returns.

A write that fails part way leaves what got as far as the disk, the same as any other half finished file. There is nothing to be done about that in one pass, since a parquet file is only a parquet file once its footer is on.

func WriteMetadata added in v0.0.18 

func WriteMetadata(w io.Writer, m *Metadata) (int64, error)

WriteMetadata writes the footer of a parquet file, and with it the last eight bytes of the file.

A parquet file is the four bytes PAR1, then the pages, then the footer, then how long the footer is in four bytes, then PAR1 again. This writes the last three of those, so a writer that has put its pages down finishes the file with one call to it and a reader working backwards from the end finds everything else. It returns how many bytes it wrote.

The footer is built whole before any of it goes out, because the length behind it is not known until it is finished. That is a few hundred bytes a column chunk, which is what a footer is.

Nothing here checks that the offsets in the metadata point at anything. A footer is a description of a file and this writes the description it was given, so a caller that hands it offsets from one file and pages from another gets a file that opens and reads nonsense. The whole file writer is what keeps those two in step.

func WritePage added in v0.0.19 

func WritePage(w io.Writer, h *PageHeader, body []byte) (int64, error)

WritePage writes a page: its header, then the body behind it.

The body is the bytes as they go in the file, which means already encoded, already compressed if the chunk is compressed, and with the levels in front of the values. Nothing here encodes or compresses anything, and the body is written from where it sits rather than copied. It returns how many bytes it wrote, header and body together, which is what a caller adds up to know where the next page starts.

The header has to agree with the body. CompressedSize is the length of the body, and for the second version of the data page the two level lengths are bytes of the body and come out of the front of it, so the three together are a claim this checks rather than takes. It asks the same things of the header that the reader asks of one it read, so a page written here is a page this package would accept. A header that does not agree with its body is refused and nothing is written.

The checksum is the one field the caller does not fill in. HasCRC says whether to write one and the value is computed here over the body, so a page this writes can never carry a checksum that disagrees with it. The CRC field of the header is ignored on the way out.

Types

type Batch added in v0.0.12 

type Batch struct {
	// Length is the number of rows.
	Length int

	// Columns are the values, one array per field of the reader's schema, in
	// the order the projection named them.
	Columns []*array.Array
}

Batch is the columns of one row group.

type BitPackedDecoder added in v0.0.8 

type BitPackedDecoder struct {
	// contains filtered or unexported fields
}

BitPackedDecoder reads levels written in the encoding parquet calls BIT_PACKED, which is the one it deprecated and which old files still have.

It is packed values and nothing else: no runs, no counts, and no length. How many values there are is not in the data, so the decoder reads to the end of what it was given and the caller keeps the ones it asked for. That is what the format intends, since a page says how many values it holds and the levels are as long as they need to be to hold that many.

The bits go the other way up here than they do in a packed run of the encoding that replaced this one. A value is read from the top of a byte down rather than from the bottom up, which is the only difference between the two and is enough to turn every value into a different one.

A width of nought reads no values, since a value of no bits leaves nothing to count them with. Levels that wide belong to a required column and are not written down at all.

The zero value is a decoder of no values. Use NewBitPackedDecoder or Reset.

func NewBitPackedDecoder added in v0.0.8 

func NewBitPackedDecoder(data []byte, width int) (*BitPackedDecoder, error)

NewBitPackedDecoder returns a decoder reading width bit values out of data.

func (*BitPackedDecoder) Read added in v0.0.8 

func (d *BitPackedDecoder) Read(dst []int32) (int, error)

Read decodes values into dst and returns how many it wrote. It returns io.EOF once the data has been read to the end.

func (*BitPackedDecoder) Reset added in v0.0.8 

func (d *BitPackedDecoder) Reset(data []byte, width int) error

Reset points the decoder at other bytes.

type BloomFilter added in v0.0.13 

type BloomFilter struct {
	// contains filtered or unexported fields
}

BloomFilter is the filter a writer wrote for one column chunk.

It says no for certain and yes with a chance of being wrong, so a scan uses it to skip and never to answer: a chunk the filter kept still has to be read and looked at. How often it is wrong is what the writer chose when it wrote the thing, and nothing in the file says what that was.

func ReadBloomFilter added in v0.0.13 

func ReadBloomFilter(r io.ReaderAt, size int64, c *ColumnChunk) (*BloomFilter, error)

ReadBloomFilter reads the filter a writer wrote for one column chunk.

The size is the size of the file, the same one ReadMetadata was given, since where the filter is and how long it is are numbers out of a footer.

It comes back nil when the chunk has no filter, which is most chunks: a writer writes them for the columns it is told to and no writer is told to by default, the bitset costing bytes in the file for a saving that only some scans take. That is not an error, and a caller that gets nil is left with the bounds.

A filter written with an algorithm, a hash or a compression this package does not know is refused rather than read anyway. The format has one of each so far, so this is a file from a future nobody has written yet.

func (*BloomFilter) Bytes added in v0.0.13 

func (f *BloomFilter) Bytes() int

Bytes is the size of the bitset, which is the only thing a filter says about how much it can be trusted. A writer sizes it for the values it expected and the error rate it wanted, so a bigger one over the same chunk is wrong less often, and neither number is written down anywhere.

func (*BloomFilter) Has added in v0.0.13 

func (f *BloomFilter) Has(value []byte) bool

Has says whether the chunk may hold the value.

False is certain and true is not. A value that was written set the bits this looks at, so a clear bit means the chunk never held it and the whole chunk can be skipped. A set one means the value is there or something else set the same bits, which is what a scan reads the chunk to find out.

The value is one value written the way a page writes it, which is the same bytes a bound in Statistics holds: a number little endian in its four or eight or twelve bytes, and a byte array as itself with no length in front of it. A caller with a Go value turns it into those bytes the way the column's type says, since the filter was built on what the file holds rather than on what a reader would rather have.

A filter that is nil rules nothing out and says yes to everything. That is what a chunk without a filter should do, and it is what ReadBloomFilter hands back for one, so a scan can ask without looking first.

func (*BloomFilter) HasString added in v0.0.13 

func (f *BloomFilter) HasString(value string) bool

HasString is Has for a value already in hand as a string, which is what a filter on a column of names carries. It is the same lookup and copies nothing.

type BoundaryOrder added in v0.0.13 

type BoundaryOrder int32

BoundaryOrder is whether the pages of a column chunk are in order, which is what lets a scan stop looking rather than look at every page.

A writer that wrote its rows sorted says so here, and a reader looking for one value in an ascending column can find the page it would be in by halving rather than by walking. A writer that did not sort says unordered, which is the zero value and what nearly every file holds. 

const (
	Unordered BoundaryOrder = iota
	Ascending
	Descending
)

The boundary orders.

func (BoundaryOrder) String added in v0.0.13 

func (o BoundaryOrder) String() string

String returns the name the format gives the order, lowercased.

type Bounds added in v0.0.12 

type Bounds struct {
	// Values holds the smallest value of the chunk and then the largest, so it
	// is an array of two of whatever the column holds. It is nil when the chunk
	// said nothing about its range, or said something this package will not act
	// on, and a caller that finds it nil has to read the chunk.
	Values *array.Array

	// MinExact and MaxExact say the bounds are values out of the chunk rather
	// than values either side of it. A writer that cut a long string down to
	// keep the footer small says so this way, and the bound is still a bound:
	// an inexact minimum is below every value in the chunk rather than equal to
	// one of them, which is all a skip needs. What it cannot do is answer a
	// question about the value itself, so a scan reading a minimum out of the
	// footer instead of the column wants an exact one.
	MinExact bool
	MaxExact bool

	// Count is how many values the chunk holds, which for a flat column is how
	// many rows the row group has. Nulls is how many of them are missing and
	// means nothing unless HasNulls, since a writer that said nothing and a
	// writer that said none are not saying the same thing.
	Count    int64
	Nulls    int64
	HasNulls bool
}

Bounds is what a writer said about the values of one column chunk.

func ReadBounds added in v0.0.12 

func ReadBounds(c Column, m *ColumnMeta) (Bounds, error)

ReadBounds decodes what a writer said about the values of one column chunk.

The chunk has to be the one holding the column c, which is one of the leaves Metadata.Columns returned, and the order the file gave that column comes along on it. Nothing is read out of the file: the bounds are in the footer, which is the whole point of them.

The bounds come back decoded into an array of two values, the smallest first, or as no array at all when the chunk said nothing worth acting on. That covers a chunk with no statistics, one bounded only by the pair the format deprecated on a type whose old order was the wrong one, a column whose type has no order at all, and a float chunk bounded by a NaN.

A bound this package cannot decode is an error rather than an absence, the same way a column it cannot assemble is. A chunk whose bounds contradict what it says about itself is an error too, since a footer that disagrees with itself is not one to skip anything on.

func (Bounds) AllNull added in v0.0.12 

func (b Bounds) AllNull() bool

AllNull says every value of the chunk is missing.

It is worth its own question because it is the one thing a filter can settle without comparing anything: a chunk of nothing but nulls has no value that matches anything, so a filter of any kind skips the row group. A chunk of no values at all counts as one, since it has nothing to match either, and it says so whether or not the writer bothered to count the nulls of a chunk it wrote nothing in.

type Codec 

type Codec int32

Codec is how the pages of a column chunk are compressed. It is per chunk rather than per file, so one file may hold a snappy column and a zstd one. 

const (
	Uncompressed Codec = iota
	Snappy
	Gzip
	LZO
	Brotli
	LZ4
	Zstd
	LZ4Raw
)

The compression codecs.

func (Codec) String 

func (c Codec) String() string

String returns the name the format gives the codec, lowercased.

type Column added in v0.0.7 

type Column struct {
	// Path is the names from the root down to the leaf, without the root's
	// own. It is what a row group's chunks are keyed by and what a projection
	// names.
	Path []string

	// Element is the leaf itself: its physical type, its width when that type
	// is a fixed length byte array, and its annotation.
	Element SchemaElement

	// Type is what one value means in kuma's types. It is the type of a value
	// and not of the column: a leaf inside a repeated group is an int64 here
	// and a list of int64 in the schema the file adds up to.
	Type dtype.DataType

	// MaxDefinition is the definition level of a value that is present all the
	// way down its path, and MaxRepetition is the repetition level inside the
	// innermost repeated group above it.
	//
	// The two of them are how a flat run of values becomes nulls and list
	// boundaries again. A value whose definition level is below MaxDefinition
	// is missing, and which of the optional nodes on the path it is missing at
	// is how far below. A MaxRepetition of zero means the column has exactly
	// one value per row and the file writes no repetition levels for it at
	// all, which is the case for every column of a flat table.
	MaxDefinition int
	MaxRepetition int

	// Order is how the file said the column's values compare, which is what
	// the bounds on its chunks mean. It is UndefinedOrder when the file did not
	// say and in a Column built by hand, and either way that is what makes
	// ReadBounds fall back to the pair of bounds that predate the question.
	Order ColumnOrder
}

Column is a leaf of the schema, which is one column of values in the file.

func (*Column) Name added in v0.0.7 

func (c *Column) Name() string

Name returns the column's path joined with dots, which is what the column is called everywhere outside the schema.

type ColumnChunk 

type ColumnChunk struct {
	// FilePath is the file the chunk lives in, which is empty for every file
	// written this decade. It is how the format allowed a single logical file
	// to be spread across several, which nothing does any more.
	FilePath string

	// FileOffset is where the chunk's metadata is, in the files that repeat it
	// next to the data. It is zero in most files.
	FileOffset int64

	// Meta is the chunk itself: where its pages are and what is in them.
	Meta ColumnMeta

	// ColumnIndex and OffsetIndex are two more structures at the end of the
	// file, holding the smallest and largest value of every page and where
	// every page starts. They are what makes skipping work at page granularity
	// rather than at row group granularity. Both offsets are zero when the
	// writer did not produce them.
	ColumnIndexOffset int64
	ColumnIndexLength int32
	OffsetIndexOffset int64
	OffsetIndexLength int32
}

ColumnChunk is one column of one row group.

func (*ColumnChunk) Start added in v0.0.7 

func (c *ColumnChunk) Start() int64

Start is where a column chunk begins in the file.

It is the dictionary page when there is one and the first data page when there is not, worked out from the two page offsets rather than read from the chunk's own FileOffset. That field is meant to say this and enough writers have got it wrong over the years that no reader trusts it.

Zero is either offset saying it is not there, which the format leaves to a reader to work out and which is safe to read that way because no page can live at nought: the first four bytes of the file are the magic. A chunk of a file with no rows in it has a dictionary page and no data page at all, and says so with a data page offset of zero.

type ColumnIndex added in v0.0.13 

type ColumnIndex struct {
	// NullPages says, per page, that every value in it is missing. The format
	// leaves the bounds of such a page undefined, so this is what has to be
	// read before them.
	NullPages []bool

	// Min and Max are the smallest and largest value of each page, in the order
	// the format defines for the column's type. There is no saying whether they
	// are values out of the page or values either side of it, which Statistics
	// does say, so a bound out of here is treated as inexact.
	Min [][]byte
	Max [][]byte

	// Order is whether the pages themselves are in order.
	Order BoundaryOrder

	// NullCounts is how many values of each page are missing, and is empty in a
	// file that did not count them.
	NullCounts []int64
}

ColumnIndex is what a writer said about the values of every page of one column chunk.

The three lists are parallel and are as long as the chunk has pages. The values are the raw bytes of the physical type, the same as in Statistics, and PageBounds is what decodes them.

func ReadColumnIndex added in v0.0.13 

func ReadColumnIndex(r io.ReaderAt, size int64, c *ColumnChunk) (*ColumnIndex, error)

ReadColumnIndex reads the bounds of every page of a column chunk.

The size is the size of the file, the same one ReadMetadata was given, since the offset in the chunk is a number out of a footer and a claim to be past the end of the file must not turn into a read of that size.

It comes back nil when the writer wrote no column index, which is most files written before the format had one and any file whose writer was not asked for it. That is not an error: a scan without a page index skips row groups and reads all of the ones it keeps, which is where the reader was before.

type ColumnMeta 

type ColumnMeta struct {
	Type Type

	// Encodings are every encoding used by any page of the chunk, which is
	// more than one whenever there is a dictionary, since the dictionary page
	// and the data pages are not encoded the same way.
	Encodings []Encoding

	// Path is the column's name, one element per level of the schema, so a
	// nested field is address.city rather than city.
	Path []string

	Codec Codec

	// NumValues counts the values in the chunk, which for a column inside a
	// repeated field is more than the number of rows.
	NumValues int64

	TotalUncompressedSize int64
	TotalCompressedSize   int64

	// DataPageOffset is where the first data page starts, and
	// DictionaryPageOffset is where the dictionary page starts, or zero when
	// the chunk has none. A chunk with a dictionary starts at the dictionary
	// page, which comes before the data pages.
	DataPageOffset       int64
	DictionaryPageOffset int64

	// IndexPageOffset is a feature the format defined and no writer produces.
	IndexPageOffset int64

	// Stats is what the writer said about the values, which is the other half
	// of skipping a row group.
	Stats Statistics

	// BloomFilterOffset is where the chunk's bloom filter is, or zero when it
	// has none. A bloom filter answers whether a value is definitely not in
	// the chunk, which is what makes an equality filter skip a group whose
	// range happens to contain the value.
	BloomFilterOffset int64
	BloomFilterLength int32

	// PageStats counts the pages of the chunk by what they hold and how they
	// are encoded, and is empty for a writer that did not produce it.
	PageStats []PageEncodingStats

	// Sizes is what the chunk costs to hold once it is decoded, and is what a
	// reader would otherwise have to read the pages to find out.
	Sizes SizeStatistics
}

ColumnMeta is where a column chunk's pages are and what is in them.

type ColumnOrder added in v0.0.12 

type ColumnOrder int32

ColumnOrder is how the values of a column compare, which is the only thing that makes the smallest and largest value of a chunk mean anything.

The format has one order and a way of saying it has none. A column the file gave TypeDefinedOrder is ordered the way the format defines for its type, which is signed for a signed integer, unsigned for an unsigned one, and byte by byte for a string. A column the file said nothing about is ordered however the writer felt like ordering it, and the bounds on its chunks are one writer's opinion rather than something a reader can act on. 

const (
	UndefinedOrder ColumnOrder = iota
	TypeDefinedOrder
)

The column orders. Undefined is the zero value because a file that says nothing is the case a reader has to handle, and because an order this package has never heard of is one it knows nothing about either.

func (ColumnOrder) String added in v0.0.12 

func (o ColumnOrder) String() string

String returns the name the format gives the order, lowercased.

type ColumnReader added in v0.0.9 

type ColumnReader struct {
	// contains filtered or unexported fields
}

ColumnReader assembles the pages of one column chunk into an array.

Pages are handed to it in the order the file has them and it hands back an array at the end. It holds one builder and one set of buffers for the whole chunk, so a chunk of a hundred pages allocates what one page needs and reuses it.

The reader is for one column, decided when it is made. What it can read is what the page decoders can read: a flat column, written plainly, as indices into a dictionary or as differences of one of the three kinds. Anything else is refused rather than guessed at. It reads a page that has already had its compression undone, which is what Chunk uses a Decompressor for.

func NewColumnReader added in v0.0.9 

func NewColumnReader(c Column) (*ColumnReader, error)

NewColumnReader returns a reader for the column c.

func (*ColumnReader) Chunk added in v0.0.12 

func (r *ColumnReader) Chunk(src io.ReaderAt, size int64, chunk *ColumnChunk) (*array.Array, error)

Chunk reads one column chunk of a file into an array.

The size is the size of the file, the same one ReadMetadata was given, and the chunk has to be one of this column's. A reader is made for one column and knows what that column's pages should hold, so handing it another one's chunk is a way of reading the wrong values without being told.

A reader is good for one chunk after another, which is what makes it worth keeping while a scan walks the row groups of a file: the builder and the buffers a column is assembled in are made once rather than once per row group. A reader that returned an error stopped somewhere inside a chunk and is holding half a column, so it is not worth handing another one.

The pages have their compression undone on the way through, which is what a Decompressor is for. The codec is a property of the chunk, so a column compressed one way in one row group and another way in the next is read the way each of them was written.

func (*ColumnReader) DType added in v0.0.9 

func (r *ColumnReader) DType() dtype.DataType

DType returns the type of the values of the column being read.

A chunk written as indices into a dictionary comes back dictionary encoded, so what Finish hands back for one of those is a dictionary of this type rather than this type. Which of the two shapes a chunk has is not known until its pages have been read, since a chunk that fills its dictionary and writes the rest of itself plainly comes back as this type after all.

func (*ColumnReader) Finish added in v0.0.9 

func (r *ColumnReader) Finish() (*array.Array, error)

Finish returns the values assembled so far and leaves the reader ready for another chunk of the same column.

A chunk that was written as indices into a dictionary comes back dictionary encoded rather than expanded, so a column of a million rows holding two hundred distinct strings is a million indices and two hundred strings. That is the shape it was written in and the shape the kernels would rather have it in, and expanding it would be undoing the one thing the encoding is for.

The exception is a chunk that gave its dictionary up part way through, which expand has already turned back into values by the time this is reached.

func (*ColumnReader) Len added in v0.0.9 

func (r *ColumnReader) Len() int

Len returns how many values have been assembled, nulls included.

func (*ColumnReader) Page added in v0.0.9 

func (r *ColumnReader) Page(p Page) error

Page assembles one page.

The body is the page as it sits in the file with whatever compression the chunk used already undone, which is what Chunk uses a Decompressor for. The levels are still in front of the values, since where they are depends on which version of the data page it is and that is this function's business rather than its caller's.

type ConvertedType 

type ConvertedType int32

ConvertedType is what a physical type means, as parquet wrote it before logical types existed. A writer that wants to be read by everything writes both, so this is usually the same thing the logical type says and is the only thing an old file says at all. 

const (
	NoConverted ConvertedType = iota - 1
	ConvertedUTF8
	ConvertedMap
	ConvertedMapKeyValue
	ConvertedList
	ConvertedEnum
	ConvertedDecimal
	ConvertedDate
	ConvertedTimeMillis
	ConvertedTimeMicros
	ConvertedTimestampMillis
	ConvertedTimestampMicros
	ConvertedUint8
	ConvertedUint16
	ConvertedUint32
	ConvertedUint64
	ConvertedInt8
	ConvertedInt16
	ConvertedInt32
	ConvertedInt64
	ConvertedJSON
	ConvertedBSON
	ConvertedInterval
)

The converted types.

func (ConvertedType) String 

func (c ConvertedType) String() string

String returns the name the format gives the converted type, lowercased.

type Decompressor added in v0.0.11 

type Decompressor struct {
	// contains filtered or unexported fields
}

Decompressor undoes the compression of the pages of one column chunk.

It holds the buffer the pages come back in, so a scan reading a chunk of a thousand pages allocates once rather than once per page. That is also what a caller has to know about it: a page it hands back holds until the next one is asked for, in the way a bufio.Scanner's token does.

The zero value undoes nothing, which is what a chunk that was not compressed wants. Use NewDecompressor.

func NewDecompressor added in v0.0.11 

func NewDecompressor(c Codec) (*Decompressor, error)

NewDecompressor returns a decompressor for a chunk written with the codec, and refuses a codec this package cannot undo.

The refusal is here rather than at the first page so that a scan finds out what it cannot read before it reads anything.

func (*Decompressor) Codec added in v0.0.11 

func (d *Decompressor) Codec() Codec

Codec is the codec the decompressor undoes.

func (*Decompressor) Page added in v0.0.11 

func (d *Decompressor) Page(p Page) (Page, error)

Page returns the page with its body decompressed.

The body it comes back with is the levels and the values together, the way an uncompressed page holds them, so what reads a page does not have to ask which codec it went through. The header is left as the file wrote it, which means the two sizes in it still say what the page took on disk and what it comes to, and it is the second one that says how long the body now is.

A page of a chunk that was not compressed comes back as it went in, pointing into the bytes of the chunk. Everything else points into the decompressor and holds only until the next page is asked for.

type DeltaByteArrayDecoder added in v0.0.10 

type DeltaByteArrayDecoder struct {
	// contains filtered or unexported fields
}

DeltaByteArrayDecoder reads values written in the encoding parquet calls DELTA_BYTE_ARRAY, which is how much of each value the one in front of it already said and then the rest of it.

The zero value is a decoder of no values. Use Reset.

func (*DeltaByteArrayDecoder) Len added in v0.0.10 

func (d *DeltaByteArrayDecoder) Len() int

Len returns how many values the decoder has not handed back yet.

func (*DeltaByteArrayDecoder) Read added in v0.0.10 

func (d *DeltaByteArrayDecoder) Read(dst [][]byte) (int, error)

Read hands back values into dst and returns how many it wrote. It returns io.EOF once the page has been read to the end, in the way an io.Reader does.

The values point into the decoder's own buffer and hold until the next Reset, so whatever appends them takes its own copy, which is what it would have to do for a plain page anyway.

func (*DeltaByteArrayDecoder) Reset added in v0.0.10 

func (d *DeltaByteArrayDecoder) Reset(data []byte) error

Reset points the decoder at the values of a page and puts all of them together.

A value is made of bytes from the one in front of it and bytes of its own, so it cannot be a slice of the page the way the other decoders hand values back. It is built into a buffer the decoder keeps and reuses from page to page, and building the lot here rather than a value at a time is what lets the values be slices of that buffer: appending to it moves it, and a value handed back before the next one was built would be left pointing at where it used to be.

type DeltaDecoder added in v0.0.10 

type DeltaDecoder struct {
	// contains filtered or unexported fields
}

DeltaDecoder reads values written in the encoding parquet calls DELTA_BINARY_PACKED.

The header of a page says how the blocks in it are cut up, so the decoder has to read that before it knows anything, which is why Reset returns an error where the other decoders here do not.

The zero value is a decoder of no values. Use NewDeltaDecoder or Reset.

func NewDeltaDecoder added in v0.0.10 

func NewDeltaDecoder(data []byte) (*DeltaDecoder, error)

NewDeltaDecoder returns a decoder reading the values in data.

The data is the values of a page and nothing else, the way PlainDecoder wants them, with the levels in front of them already taken off.

func (*DeltaDecoder) Len added in v0.0.10 

func (d *DeltaDecoder) Len() int

Len returns how many values the decoder has not handed back yet, which after Reset is how many the page said it holds.

func (*DeltaDecoder) Offset added in v0.0.10 

func (d *DeltaDecoder) Offset() int

Offset returns how many bytes of the page the decoder has read.

It is what the two byte array encodings need. Both of them put a block of this encoding in front of the bytes it describes, and nothing says where the block ends but reading it, since how many bytes it takes follows from the widths inside it. So this means what it says once Len has come down to nought and rather less before then.

func (*DeltaDecoder) Read added in v0.0.10 

func (d *DeltaDecoder) Read[T deltaValue](dst []T) (int, error)

Read decodes values into dst and returns how many it wrote. It returns io.EOF once the page has been read to the end, in the way an io.Reader does.

How many values a page holds is in its header rather than in how many bytes it takes, so this stops at that count and never at the end of the data. The bytes after it are the padding of the last miniblock.

func (*DeltaDecoder) Reset added in v0.0.10 

func (d *DeltaDecoder) Reset(data []byte) error

Reset points the decoder at other bytes and reads the header in front of them, so that a scan reading a thousand pages of one column does not allocate a decoder for each of them.

The header is how big a block is, how many miniblocks are in one, how many values the page holds and the first of those values. Everything after it is differences, and the first value is the only one written down as it is.

type DeltaLengthDecoder added in v0.0.10 

type DeltaLengthDecoder struct {
	// contains filtered or unexported fields
}

DeltaLengthDecoder reads values written in the encoding parquet calls DELTA_LENGTH_BYTE_ARRAY, which is the lengths of the values written as differences and then the bytes of all of them end to end.

The zero value is a decoder of no values. Use Reset.

func (*DeltaLengthDecoder) Len added in v0.0.10 

func (d *DeltaLengthDecoder) Len() int

Len returns how many values the decoder has not handed back yet.

func (*DeltaLengthDecoder) Read added in v0.0.10 

func (d *DeltaLengthDecoder) Read(dst [][]byte) (int, error)

Read hands back values into dst and returns how many it wrote. It returns io.EOF once the page has been read to the end, in the way an io.Reader does.

The values point into the page rather than into a copy, the same way a plain page's values do, so whatever appends them takes its own copy.

func (*DeltaLengthDecoder) Reset added in v0.0.10 

func (d *DeltaLengthDecoder) Reset(data []byte) error

Reset points the decoder at the values of a page and reads the lengths in front of them.

The lengths are checked against the bytes here rather than as each value is handed back, since a page whose bytes run out short of what its lengths ask for is one thing to say once and not once per value. A page this refuses leaves a decoder of no values rather than of the ones it had got to, which is the only answer that means anything for half a page.

type Encoding 

type Encoding int32

Encoding is how the values of a page are written down. A column chunk lists every encoding it used, since the dictionary page and the data pages of one chunk are not encoded the same way. 

const (
	NoEncoding           Encoding = -1
	Plain                Encoding = 0
	PlainDictionary      Encoding = 2
	RLE                  Encoding = 3
	BitPacked            Encoding = 4
	DeltaBinaryPacked    Encoding = 5
	DeltaLengthByteArray Encoding = 6
	DeltaByteArray       Encoding = 7
	RLEDictionary        Encoding = 8
	ByteStreamSplit      Encoding = 9
)

The encodings. Two of them are numbered out of order because the format replaced them: PlainDictionary became RLEDictionary and BitPacked became RLE, and files written before the change still say the old thing.

NoEncoding is a page that did not say how it was encoded. The format calls that field required, so it is a file to refuse rather than a page to guess at, and it needs a value of its own because zero is a real encoding.

func (Encoding) String 

func (e Encoding) String() string

String returns the name the format gives the encoding, lowercased.

type FileReader added in v0.0.12 

type FileReader struct {
	// contains filtered or unexported fields
}

FileReader reads the row groups of a parquet file, a projection at a time.

It is made from the footer and keeps it, so making one reads the end of the file and nothing else. What it hands back is a Batch per row group holding the columns Project named, which is all of them until it is called.

It keeps the buffers a column is assembled in from one row group to the next, so a scan of a file of a thousand row groups allocates what one of them needs rather than a thousand times that. That is also what makes it a thing to use from one goroutine at a time: two goroutines reading row groups of the same file want a reader each, which costs another footer and nothing more.

func NewFileReader added in v0.0.12 

func NewFileReader(r io.ReaderAt, size int64) (*FileReader, error)

NewFileReader returns a reader for the parquet file in r.

The size is the size of the file, the same one ReadMetadata takes and for the same reason: a footer at the end of a file cannot be found by anything that only reads forwards. Nothing but the footer is read here.

The reader starts with every column of the file projected, in the order the row groups hold their chunks. Project is what narrows it.

func (*FileReader) BloomFilter added in v0.0.13 

func (r *FileReader) BloomFilter(group, column int) (*BloomFilter, error)

BloomFilter returns the filter the writer wrote for one projected column of one row group.

The column is its place in the projection rather than in the file, the same as Bounds and PageBounds. It comes back nil when that chunk has no filter, which is what a scan falls back to the bounds on.

func (*FileReader) Bounds added in v0.0.12 

func (r *FileReader) Bounds(i int) ([]Bounds, error)

Bounds returns what the writer said about the projected columns of one row group, one entry per column and in the order they were projected.

This is what a row group is skipped on. The bounds are in the footer, so nothing is read out of the file to answer it: a scan asks each group what its columns hold, works out that nothing in it can match, and moves on without touching a page. ReadBounds is what each entry comes from and says what is in one and how much of it is worth acting on.

A column that says nothing about itself comes back with no bounds rather than as an error, since a writer is allowed to write no statistics and most of the old ones wrote none worth reading. What is an error is a footer that contradicts itself, the same as it is for reading the values.

func (*FileReader) BytesRead added in v0.0.12 

func (r *FileReader) BytesRead() int64

BytesRead returns how many bytes have come out of the file through this reader so far, the footer included.

This is what says a projection worked. Reading two columns of a file of two hundred has to cost what those two columns take, and the way to know it did is to add up the reads rather than to trust that the offsets in the footer were used.

func (*FileReader) Columns added in v0.0.12 

func (r *FileReader) Columns() []Column

Columns returns every leaf of the file's schema, whatever is projected.

These are what Project names, in the order a row group holds its chunks, which is the order a projection of all of them comes back in.

func (*FileReader) Metadata added in v0.0.12 

func (r *FileReader) Metadata() *Metadata

Metadata returns the footer the reader works from.

It is the reader's own rather than a copy of it, so a caller that changes it changes what the next read reads.

func (*FileReader) NumRowGroups added in v0.0.12 

func (r *FileReader) NumRowGroups() int

NumRowGroups returns how many row groups the file holds.

func (*FileReader) NumRows added in v0.0.12 

func (r *FileReader) NumRows() int64

NumRows returns how many rows the file holds, whatever is projected.

func (*FileReader) PageBounds added in v0.0.13 

func (r *FileReader) PageBounds(group, column int) ([]PageBounds, error)

PageBounds returns what the writer said about each page of one projected column of one row group.

The column is its place in the projection rather than in the file, the same as the entries Bounds hands back, so a reader projecting two columns is asked about page bounds one column at a time and not once for the pair. That is because reading them costs a read of the file: the indexes are not in the footer, and a scan filtering on one column has no reason to read the index of the other.

It comes back nil when the writer wrote no page index for the chunk, and a caller that gets nil is where FileReader.Bounds left it, which is a row group it either reads or skips whole.

func (*FileReader) Project added in v0.0.12 

func (r *FileReader) Project(names ...string) error

Project narrows what a batch holds to the named columns, in the order they are named.

A name is what Column.Name returns, which is the column's path joined with dots, so a leaf inside a group is "point.x" rather than "x". A name the file does not have is an error and nothing is narrowed, since a projection that was half applied would hand back batches nobody asked for.

Naming a column twice reads it twice, which is a waste rather than a mistake. Naming none of them narrows the reader to no columns at all, which is what counting the rows of a file costs nothing with. Projecting again starts from the file's own columns rather than from what is projected now, so it widens as easily as it narrows.

func (*FileReader) RowGroup added in v0.0.12 

func (r *FileReader) RowGroup(i int) (Batch, error)

RowGroup reads the projected columns of one row group.

The columns are read in the order they were projected, one chunk each, and the batch comes back holding all of them. Reading the same row group twice reads the file twice: nothing is cached, since a row group of a real file is tens of megabytes and a caller that wants it twice can keep it.

func (*FileReader) RowGroups added in v0.0.16 

func (r *FileReader) RowGroups(filter ...Predicate) ([]int, error)

RowGroups returns the row groups that may hold a row passing every predicate, in the order the file holds them.

This is the pushdown. Every group it leaves out is one whose statistics say it holds no matching row, which the reader worked out from the footer and, where the writer wrote one, from a bloom filter. A group it returns is one that may hold a matching row rather than one that does, so the rows still have to be filtered once they are read.

With no predicates it returns every group, which is what an unfiltered scan does and costs nothing to ask for. A predicate naming a column the file does not have is an error, the same as a projection naming one, since a filter that was quietly dropped would read the whole file and look like it worked.

It reads the footer, which is already in hand, and a bloom filter for each equality on a column that has one. A file whose writer wrote no bloom filters, which is nearly all of them, is answered without touching the file at all.

Example

Reading the row groups a filter cannot rule out and leaving the rest of the file alone. 

package main

import (
	"bytes"
	"fmt"
	"log"
	"os"

	"github.com/tamnd/kuma/kernel"
	"github.com/tamnd/kuma/parquet"
)

// openTestdata makes a reader for one of the files the tests are run against.
func openTestdata(name string) *parquet.FileReader {
	buf, err := os.ReadFile("testdata/" + name)
	if err != nil {
		log.Fatal(err)
	}
	r, err := parquet.NewFileReader(bytes.NewReader(buf), int64(len(buf)))
	if err != nil {
		log.Fatal(err)
	}
	return r
}

func main() {
	r := openTestdata("stats.parquet")

	// The file holds twelve rows in three row groups with n running from nought
	// to eleven, so two of the three cannot hold a row of eight or more and the
	// footer already says which.
	groups, err := r.RowGroups(parquet.Where("n", kernel.OpGe, int64(8)))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("reading", len(groups), "of", r.NumRowGroups(), "row groups")

	for _, g := range groups {
		b, err := r.RowGroup(g)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println("row group", g, "holds", b.Length, "rows")
	}
}

Output:
reading 1 of 3 row groups
row group 2 holds 4 rows
Example (Bloom)

Ruling out a value that is inside the range of every row group, which is what a writer writes a bloom filter for. 

package main

import (
	"bytes"
	"fmt"
	"log"
	"os"

	"github.com/tamnd/kuma/kernel"
	"github.com/tamnd/kuma/parquet"
)

// openTestdata makes a reader for one of the files the tests are run against.
func openTestdata(name string) *parquet.FileReader {
	buf, err := os.ReadFile("testdata/" + name)
	if err != nil {
		log.Fatal(err)
	}
	r, err := parquet.NewFileReader(bytes.NewReader(buf), int64(len(buf)))
	if err != nil {
		log.Fatal(err)
	}
	return r
}

func main() {
	r := openTestdata("bloom.parquet")

	// The identifiers go up in sevens, so 1004 sits between two of them: inside
	// the bounds of the first group and not in the file.
	for _, id := range []int64{1007, 1004} {
		groups, err := r.RowGroups(parquet.Where("id", kernel.OpEq, id))
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(id, "may be in row groups", groups)
	}
}

Output:
1007 may be in row groups [0]
1004 may be in row groups []

func (*FileReader) Schema added in v0.0.12 

func (r *FileReader) Schema() dtype.Schema

Schema returns what a batch holds, which is one field per projected column.

The fields are the leaves of the file's schema and are named the way a projection names them, so a leaf inside a group is "point.x" here and is a field of a field in the schema the file itself describes, which is what Metadata.Schema returns. A column is nullable when anything on its path is optional.

The file's key and value metadata comes back on it whatever is projected, since it describes the file rather than the columns. That includes the Arrow schema pyarrow and Spark write under ARROW:schema, which still names every column of the file and not the projected ones.

A chunk written as indices into a dictionary comes back dictionary encoded, which ColumnReader.Finish says more about, so the type of a column in a batch is either the type of the field here or a dictionary of it.

type KeyValue 

type KeyValue struct {
	Key   string
	Value string
}

KeyValue is one entry of the metadata a writer attached to a file.

type LogicalKind 

type LogicalKind int32

LogicalKind is what a physical type means, as parquet writes it now. It is the same idea as a converted type with the parameters that one could not carry: a decimal says its precision here rather than in the schema element, and a timestamp says whether it is UTC. 

const (
	NoLogical LogicalKind = iota
	StringLogical
	MapLogical
	ListLogical
	EnumLogical
	DecimalLogical
	DateLogical
	TimeLogical
	TimestampLogical
	IntegerLogical
	UnknownLogical
	JSONLogical
	BSONLogical
	UUIDLogical
	Float16Logical
)

The logical types, in the order the union declares them. The numbers are this package's own rather than the format's, which are the field numbers of the union and have a gap in them where an interval type was reserved and never defined.

func (LogicalKind) String 

func (l LogicalKind) String() string

String returns the name the format gives the logical type, lowercased.

type LogicalType 

type LogicalType struct {
	Kind LogicalKind

	Scale     int32
	Precision int32

	// UTC says the writer knew what instant the value stands for. A timestamp
	// that is not adjusted to UTC is a wall clock reading with no zone, which
	// is a different thing from a timestamp in UTC.
	UTC  bool
	Unit TimeUnit

	BitWidth int8
	Signed   bool
}

LogicalType is what a physical type means.

It is a union in the format, so which of the fields below mean anything depends on Kind. A DecimalLogical has Scale and Precision, a TimeLogical and a TimestampLogical have Unit and UTC, an IntegerLogical has BitWidth and Signed, and the rest are the kind and nothing else.

type Metadata 

type Metadata struct {
	// Version is the format version the writer wrote, which is 1 for every
	// file in the wild and 2 for files using the newer page header.
	Version int32

	// Nodes is the schema as a flattened tree, the root first, each node
	// followed by its children. A node with no children is a column and a node
	// with children is a group. The format calls this the schema, and it is
	// called something else here because the schema of a file is a tree of
	// kuma types and this is the bytes it is built out of. Tree, Columns and
	// Schema are the three ways to read it.
	Nodes []SchemaElement

	// NumRows is how many rows the whole file holds.
	NumRows int64

	// RowGroups are the horizontal slices of the file, in the order they were
	// written, which is the order the rows are in.
	RowGroups []RowGroup

	// KeyValue is what the writer attached to the file. Arrow puts its own
	// schema in here under the key ARROW:schema, which is how a file written
	// from Arrow remembers a type parquet has no way to write down.
	KeyValue []KeyValue

	// CreatedBy is the writer's name and version, as free text. It is worth
	// keeping because the format has had bugs that are identified by which
	// writer produced the file and nothing else.
	CreatedBy string

	// Orders is how the values of each column compare, one entry per leaf of
	// the schema in the order Columns hands them back. It is empty in a file
	// that did not say, and a file that did not say has left the bounds on its
	// chunks meaning nothing, which is why Column carries the order along and
	// ReadBounds asks for it.
	Orders []ColumnOrder
}

Metadata is the footer of a parquet file.

func ReadMetadata 

func ReadMetadata(r io.ReaderAt, size int64) (*Metadata, error)

ReadMetadata reads the footer of a parquet file.

The file is read backwards, the way the format is meant to be read: the last eight bytes say how long the footer is and where it therefore starts, and the footer says where everything else is. Three reads, none of them near the data, which is why opening a parquet file over a network is cheap and opening a CSV file is not.

It takes an io.ReaderAt and the size the way archive/zip does, since a footer at the end of a file cannot be found by anything that only reads forwards. The size has to be the real one: a size that is too small looks like a truncated file and a size that is too large looks like a file that is not parquet at all.

The strings in the result are copied out of the footer, and the statistics point into it, so a Metadata holds onto the bytes of the footer and nothing else. The file itself is not read beyond it and is not held open.

func (*Metadata) Columns added in v0.0.7 

func (m *Metadata) Columns() ([]Column, error)

Columns returns the leaves of the schema in the order the row groups hold their chunks, which is the order they appear in the flattened schema.

func (*Metadata) Schema 

func (m *Metadata) Schema() (dtype.Schema, error)

Schema returns what the file holds, in kuma's types.

The file's own metadata comes back on the schema, including the Arrow schema that pyarrow and Spark attach under ARROW:schema. That entry is left alone rather than read, so a type Arrow can hold and parquet cannot, a dictionary for instance, is still whatever parquet wrote it as.

Two things the schema cannot carry are dropped on the way. A list element and a map value may be optional in parquet and kuma's list and map types hold a type rather than a field, so their nullability goes no further than the levels on the columns, which is where a decoder reads it from anyway.

func (*Metadata) SetSchema added in v0.0.19 

func (m *Metadata) SetSchema(s dtype.Schema) error

SetSchema sets the schema of the file to s, which is what Schema reads back.

It fills in the nodes and the key value metadata and touches nothing else, so a caller building a footer sets the schema first and then adds the row groups that go with it. The order of the leaves is the order the columns appear in, which is the order every row group has to hold its chunks in.

A type parquet has no way of writing is an error naming the column and the type. A type kuma has two of and parquet has one of is written as the one: large strings and large lists lose the large, a fixed size list becomes an ordinary list, and a dictionary is written as the type of its values, since in parquet a dictionary is a decision about a page rather than a type. Those come back as what they were written as rather than as what they were.

Metadata on the schema is kept, since a footer has a place for it. Metadata on a field is dropped, since a footer has nowhere to put it. Nothing else is lost.

func (*Metadata) Tree added in v0.0.7 

func (m *Metadata) Tree() (Node, error)

Tree returns the schema as a tree, starting at the root.

The root is the file itself and its name is whatever the writer called the record type, usually schema or spark_schema. Its children are the columns of the file as somebody querying it would name them.

type Node added in v0.0.7 

type Node struct {
	SchemaElement

	// Children are the fields of a group, in the order the file wrote them,
	// which is the order their columns appear in every row group.
	Children []Node
}

Node is one node of the schema tree.

A node with no children is a column and carries a physical type. A node with children is a group and carries none, and what it means depends on how it is annotated: a list, a map, or a struct when it is not annotated at all.

func (*Node) Leaf added in v0.0.7 

func (n *Node) Leaf() bool

Leaf reports whether n is a column rather than a group.

type OffsetIndex added in v0.0.13 

type OffsetIndex struct {
	// Pages are the locations, in the order the pages are written, which is the
	// order their rows are in.
	Pages []PageLocation
}

OffsetIndex is where every page of one column chunk is.

func ReadOffsetIndex added in v0.0.13 

func ReadOffsetIndex(r io.ReaderAt, size int64, c *ColumnChunk) (*OffsetIndex, error)

ReadOffsetIndex reads where every page of a column chunk is.

It comes back nil when the writer wrote no offset index, on the same terms as ReadColumnIndex. A writer that wrote one of the two wrote both, but they are read one at a time because a filter that keeps no page of a chunk has no use for where those pages are.

type Options added in v0.0.15 

type Options struct {
	// Columns names the columns to read, in the order the table should hold
	// them, using the dotted path [Column.Name] returns. Nil reads every column
	// of the file in the order the file holds them.
	//
	// This is the projection, and it is the reason to use this format at all. A
	// file of two hundred columns keeps each of them in a run of pages of its
	// own, so naming three of them reads three runs and never touches the rest:
	// not read, not decompressed, not allocated for.
	//
	// A name the file does not have is an error, since a caller who asked for a
	// column by a name that is not there wants to hear about it rather than get
	// a table with a column missing.
	Columns []string

	// Dictionary keeps the encoding of any column the file wrote as indices
	// into a dictionary, so such a column comes back as a [dtype.Dictionary]
	// of the type the file's schema names rather than as that type.
	//
	// It is off by default because a caller who asked for a file of country
	// codes wants a column of strings, and because most writers encode nearly
	// every column whether it repeats or not, so leaving it on would mean a
	// dictionary of a million distinct values as readily as one of two hundred.
	// It is worth turning on for the columns it was meant for: a group by, a
	// join and a filter all read through the encoding, and a column of country
	// codes kept encoded is the size of a column of small integers.
	Dictionary bool

	// Filter says which rows to keep, as predicates a row has to pass every one
	// of. Nil reads every row.
	//
	// This is the other half of not reading a file and on a file written in any
	// sort of order it saves more than the projection does. A row group whose
	// statistics say it holds no matching row is never opened, so a scan of a
	// year of orders looking for one day reads one row group rather than three
	// hundred and sixty five, and [FileReader.RowGroups] is that part on its
	// own.
	//
	// The rows of the groups that are read are compared as well, so what comes
	// back is the rows that pass and not the row groups that might hold them. A
	// row whose value is missing does not pass, since nothing compares to a
	// value that is not there, and neither does a row of a group with no
	// statistics that turned out not to match.
	//
	// A predicate may name a column Columns does not. Filtering on a column and
	// reading it are different questions, and filtering on a timestamp that is
	// not wanted in the result is the ordinary case, so such a column is read to
	// compare the rows against and left out of the table.
	//
	//	t, err := parquet.Read(r, size, &parquet.Options{
	//		Columns: []string{"id", "price"},
	//		Filter:  []parquet.Predicate{parquet.Where("day", kernel.OpEq, int64(19000))},
	//	})
	Filter []Predicate
}

Options says what to read and in what shape.

The zero value reads every column of the file and gives each of them the type the file's schema names, which is what a nil pointer means as well.

type Page added in v0.0.7 

type Page struct {
	PageHeader
	Data []byte
}

Page is a page of a column chunk.

Data is the body as it sits in the file, which is compressed if the chunk is and holds the levels in front of the values either way. It points into the bytes of the chunk rather than copying out of them, so it stays valid for as long as the Pages it came from.

type PageBounds added in v0.0.13 

type PageBounds struct {
	Bounds

	Offset         int64
	CompressedSize int32
	FirstRow       int64
}

PageBounds is what the two indexes together say about one page.

The Bounds are what the page holds, the same shape ReadBounds gives for a whole chunk, with Count being the rows of the page rather than of the group. The rest is where the page is, so a scan that has decided to read it can.

func ReadPageBounds added in v0.0.13 

func ReadPageBounds(r io.ReaderAt, size int64, chunk *ColumnChunk, c Column, rows int64) ([]PageBounds, error)

ReadPageBounds reads both indexes of a column chunk and decodes what they say about each of its pages.

The rows are how many the row group holds, which is what the last page runs to. The index says where each page starts and never how long it is, so the length of a page is where the next one starts and the length of the last one is everything left of the group.

It comes back nil when the writer wrote no page index, which is what a caller falls back to ReadBounds on. A chunk with a column index and no offset index is a footer contradicting itself rather than a chunk without an index, since no writer produces one without the other.

type PageEncodingStats added in v0.0.18 

type PageEncodingStats struct {
	Kind     PageKind
	Encoding Encoding

	// Count is how many pages of the chunk are that kind in that encoding.
	Count int32
}

PageEncodingStats counts the pages of a column chunk that hold one thing and are encoded one way.

A chunk with a dictionary usually has two of these, one saying there is a single dictionary page in the plain encoding and one saying how many data pages there are in the dictionary encoding. That is enough to tell a reader whether a chunk fell back off its dictionary part way through without opening a page of it, which is the thing the encodings list on the chunk cannot say, since a list holding both the dictionary encoding and the plain one is the same list whether one page fell back or every page did. 

type PageHeader struct {
	// Kind is what the page holds.
	Kind PageKind

	// CompressedSize is how many bytes the body takes in the file, which is
	// what the next page header sits behind. UncompressedSize is how many it
	// takes once the codec of the chunk has been undone, and the two are the
	// same when the chunk is not compressed.
	CompressedSize   int32
	UncompressedSize int32

	// CRC is a CRC32 of the body as it sits in the file, and HasCRC says
	// whether the writer wrote one. Most do not.
	CRC    int32
	HasCRC bool

	// NumValues is how many values the page holds, nulls included, and for a
	// dictionary page how many entries the dictionary has.
	NumValues int32

	// Encoding is how the values are encoded. A dictionary page is plain or
	// plain dictionary, and a data page that leans on the dictionary says so
	// here rather than in the chunk.
	Encoding Encoding

	// DefinitionEncoding and RepetitionEncoding are how the levels in front of
	// the values are encoded. The second version of the data page does not
	// write them because the format fixes both at RLE, so they are filled in
	// here either way and a decoder does not have to ask which page it got.
	DefinitionEncoding Encoding
	RepetitionEncoding Encoding

	// NumNulls and NumRows are the second version of the data page only. The
	// row count is what makes it worth having: a page of a repeated column
	// holds more values than rows, and this is what lets a reader skip a page
	// without decoding its levels.
	NumNulls int32
	NumRows  int32

	// DefinitionLength and RepetitionLength are how many bytes of the body the
	// levels take, in the second version of the data page. They come first,
	// repetition then definition, and they are outside whatever compression
	// the rest of the body went through. The first version writes its levels
	// inside the compressed part and says nothing about how long they are.
	DefinitionLength int32
	RepetitionLength int32

	// Compressed says whether the values in the body went through the codec of
	// the chunk. The second version of the data page may turn compression off
	// for one page, which a writer does when the page did not get smaller.
	// Everything else is compressed if the chunk is.
	Compressed bool

	// Sorted is a dictionary page whose entries are in order, which lets a
	// reader compare dictionary indices instead of values. Writers rarely say
	// so even when it is true.
	Sorted bool

	// Stats is what the writer said about the values of this page. It is the
	// same structure the chunk carries and it is usually not written, since
	// the column index replaced it.
	Stats Statistics
}

PageHeader is the header in front of a page.

It is four structures in the format, one per page type, and one here. Which fields mean anything depends on Kind and is written next to each of them. The fields the format calls required have to be there: a data page that does not say how it is encoded is refused rather than read as though it were plain, since a page decoded with the wrong encoding is wrong data rather than an error.

type PageKind added in v0.0.7 

type PageKind int32

PageKind is what a page holds. A column chunk is a run of pages, at most one of them a dictionary and the rest of them data. 

const (
	DataPage PageKind = iota
	IndexPage
	DictionaryPage
	DataPageV2
)

The page types. The two data pages are two versions of the same thing: the second one moved the levels out of the compressed part of the page so that a reader can work out how many rows a page holds without decompressing it.

func (PageKind) String added in v0.0.7 

func (k PageKind) String() string

String returns the name the format gives the page type, lowercased.

type PageLocation added in v0.0.13 

type PageLocation struct {
	// Offset is where the page starts in the file, and CompressedSize is how
	// long it is, header and all, as it sits in the file.
	Offset         int64
	CompressedSize int32

	// FirstRow is the row the page starts at, counted from the first row of the
	// row group rather than of the file.
	FirstRow int64
}

PageLocation is where one page of a column chunk is.

type Pages added in v0.0.7 

type Pages struct {
	// contains filtered or unexported fields
}

Pages is the pages of one column chunk, in the order they were written.

func ReadPages added in v0.0.7 

func ReadPages(r io.ReaderAt, size int64, c *ColumnChunk) (*Pages, error)

ReadPages reads a column chunk of a file and returns its pages.

The size is the size of the file, the same one ReadMetadata was given. It is what the offsets in the chunk are checked against, since they are numbers out of a footer and a chunk claiming to start past the end of the file must not turn into a read of that size.

The whole chunk is read here, in one ReadAt, and the pages that come back point into it. Reading the chunk that a scan wants is the point of having read the footer first.

func (*Pages) Next added in v0.0.7 

func (p *Pages) Next() (Page, error)

Next reads the next page of the chunk. It returns io.EOF once the chunk has been walked to the end.

type PlainDecoder added in v0.0.8 

type PlainDecoder struct {
	// contains filtered or unexported fields
}

PlainDecoder reads values written in the plain encoding.

It does not know what type it is reading. A page says what its column's physical type is and the caller reads the values with the method for it, which is one method per type and no conversions: a column of int32 is read with Int32 and nothing else, because a value read at the wrong width is not a wrong value but a different value at every position after it.

The zero value is a decoder of no values. Use NewPlainDecoder or Reset.

func NewPlainDecoder added in v0.0.8 

func NewPlainDecoder(data []byte) *PlainDecoder

NewPlainDecoder returns a decoder reading the values in data.

The data is the values of a page and nothing else. The levels in front of them belong to whoever is taking the page apart, since how long they are is in the page header for one version of the page and in front of them for the other.

func (*PlainDecoder) Boolean added in v0.0.8 

func (d *PlainDecoder) Boolean(dst []bool) (int, error)

Boolean reads values written as boolean, which are one bit each and packed eight to a byte from the bottom up.

The bits run on across the whole page rather than starting again per call, so a page of three booleans is one byte with five bits in it that are not values.

func (*PlainDecoder) ByteArray added in v0.0.8 

func (d *PlainDecoder) ByteArray(dst [][]byte) (int, error)

ByteArray reads values written as byte_array, which is four bytes of length and then that many bytes.

The values point into the data rather than copying out of it, so they are good for as long as the page they were read from is.

func (*PlainDecoder) Double added in v0.0.8 

func (d *PlainDecoder) Double(dst []float64) (int, error)

Double reads values written as double.

func (*PlainDecoder) Fixed added in v0.0.8 

func (d *PlainDecoder) Fixed(dst [][]byte, width int) (int, error)

Fixed reads values written as fixed_len_byte_array, which are width bytes each with no length in front of them. The width is in the schema, since every value of the column is the same size.

Like ByteArray, the values point into the data.

func (*PlainDecoder) Float added in v0.0.8 

func (d *PlainDecoder) Float(dst []float32) (int, error)

Float reads values written as float, which is four bytes in the layout every machine that matters agrees on.

func (*PlainDecoder) Int32 added in v0.0.8 

func (d *PlainDecoder) Int32(dst []int32) (int, error)

Int32 reads values written as int32, which is every integer parquet stores in four bytes or fewer, a date, and a time of day in milliseconds.

func (*PlainDecoder) Int64 added in v0.0.8 

func (d *PlainDecoder) Int64(dst []int64) (int, error)

Int64 reads values written as int64, which is every wider integer and every timestamp the format has a logical type for.

func (*PlainDecoder) Int96 added in v0.0.8 

func (d *PlainDecoder) Int96(dst []int64) (int, error)

Int96 reads values written as int96 and returns them as nanoseconds since the epoch.

The twelve bytes are a count of nanoseconds into a day and then the Julian day it falls in, which is a day number counted from a morning in 4713 BC. The conversion is done here rather than left to the caller because the value is a timestamp and nothing else: no writer ever put anything but a timestamp in one, and the format has no way of saying what else it might be.

What the timestamp is in is another matter. Some writers wrote UTC and some wrote whatever the machine's clock said, and there is nothing in the file to tell them apart, which is why the schema gives an int96 column no zone.

func (*PlainDecoder) Left added in v0.0.8 

func (d *PlainDecoder) Left() int

Left is how many bytes have not been read. It is what a caller reading byte arrays has instead of a count, since the only way to know how many values are in a run of them is to walk it.

func (*PlainDecoder) Reset added in v0.0.8 

func (d *PlainDecoder) Reset(data []byte)

Reset points the decoder at other bytes, so that a scan reading a thousand pages of one column does not allocate a decoder for each of them.

type PlainEncoder added in v0.0.17 

type PlainEncoder struct {
	// contains filtered or unexported fields
}

PlainEncoder writes values in the plain encoding.

It does not know what type it is writing, the same way PlainDecoder does not know what it is reading. The caller writes the values with the method for the physical type of its column and there are no conversions, because a value written at the wrong width is not a wrong value but a different value at every position after it.

The zero value is an encoder with nothing in it and is ready to use.

func (*PlainEncoder) Boolean added in v0.0.17 

func (e *PlainEncoder) Boolean(vals []bool)

Boolean writes values as boolean, which are one bit each and packed eight to a byte from the bottom up.

The bits run on across the page rather than starting again per call, the way the decoder reads them, so a page of three booleans is one byte with five bits in it that are not values. Those bits are zero, which matters because a page is compared against another page in a test and not because anything reads them.

func (*PlainEncoder) ByteArray added in v0.0.17 

func (e *PlainEncoder) ByteArray(vals [][]byte)

ByteArray writes values as byte_array, which is four bytes of length and then that many bytes.

The length is four bytes because the format says it is, which puts a value of more than four gigabytes outside what this encoding can hold. Nothing that reads parquet would read one either.

func (*PlainEncoder) ByteArrayString added in v0.0.17 

func (e *PlainEncoder) ByteArrayString(vals []string)

ByteArrayString writes strings as byte_array, which is what a column of them is written as and saves turning every value into bytes on the way past.

func (*PlainEncoder) Bytes added in v0.0.17 

func (e *PlainEncoder) Bytes() []byte

Bytes returns the values written so far, which is the data of a page.

The bytes are the encoder's own and are good until the next write to it, so a caller holding on to a page after calling Reset holds a copy.

func (*PlainEncoder) Double added in v0.0.17 

func (e *PlainEncoder) Double(vals []float64)

Double writes values as double.

func (*PlainEncoder) Fixed added in v0.0.17 

func (e *PlainEncoder) Fixed(vals [][]byte)

Fixed writes values as fixed_len_byte_array, which is the bytes of each value with no length in front of them.

The width is in the schema rather than in the page, so every value has to be the width the column said and nothing here can check that: a value of the wrong length would be read back as the tail of one value and the head of the next. The caller writing the schema is the one that knows.

func (*PlainEncoder) Float added in v0.0.17 

func (e *PlainEncoder) Float(vals []float32)

Float writes values as float.

func (*PlainEncoder) Int32 added in v0.0.17 

func (e *PlainEncoder) Int32(vals []int32)

Int32 writes values as int32, which is every integer parquet stores in four bytes or fewer, a date, and a time of day in milliseconds.

func (*PlainEncoder) Int64 added in v0.0.17 

func (e *PlainEncoder) Int64(vals []int64)

Int64 writes values as int64, which is every wider integer and every timestamp the format has a logical type for.

func (*PlainEncoder) Len added in v0.0.17 

func (e *PlainEncoder) Len() int

Len is how many bytes have been written, which is what a caller watching for a page to reach its size looks at.

func (*PlainEncoder) Reset added in v0.0.17 

func (e *PlainEncoder) Reset()

Reset empties the encoder and keeps the buffer, so that a writer putting down a thousand pages of one column allocates for the largest of them rather than for each of them.

type Predicate added in v0.0.16 

type Predicate struct {
	// Column is the column to test, named the way [Column.Name] names it, so a
	// leaf inside a group is "point.x" rather than "x".
	//
	// It does not have to be a projected column. Filtering on a column and
	// reading it are different questions, and skipping nine row groups in ten on
	// a timestamp that never appears in the result is the ordinary case.
	Column string

	// Op is the comparison, with the column on the left and Value on the right,
	// so [kernel.OpLt] keeps the rows below Value.
	Op kernel.CompareOp

	// Value is what to compare against, as an array holding exactly one value
	// that is not missing. [array.Of] and [array.OfStrings] build one, and
	// [Where] and [WhereString] wrap the whole struct up.
	//
	// A value that is missing is refused rather than acted on. Nothing compares
	// to it, so a filter carrying one keeps no rows at all, which is a mistake
	// worth hearing about rather than an empty table worth returning.
	Value *array.Array
}

Predicate is one test on the values of one column.

It is a column compared against one value, which is what most of a filter on a scan is made of and what a writer's statistics can be asked about. A list of them is an and: a row group is read when every one of them may hold in it.

func Where added in v0.0.16 

func Where[T array.Numeric](column string, op kernel.CompareOp, v T) Predicate

Where returns a predicate comparing a column of numbers against v.

A column whose type carries something else, a timestamp with a unit or a dictionary of strings, needs a value of that same type and so needs the struct written out with an array of one built for it.

func WhereString added in v0.0.16 

func WhereString(column string, op kernel.CompareOp, v string) Predicate

WhereString is Where for a column of text.

func (Predicate) Keep added in v0.0.16 

func (p Predicate) Keep(b Bounds) (bool, error)

Keep reports whether a chunk holding these bounds may hold a row that passes the predicate.

True is the answer that costs a read and false is the one worth having. A chunk it keeps is one whose range overlaps what was asked for, which is not the same as one holding a matching row, so a caller still has to look at the rows. A chunk it drops holds no matching row at all.

Bounds a writer did not give, or gave in a form ReadBounds will not act on, are kept. So is a chunk whose bounds cannot be compared against the value, which is the one thing here that is an error, since a filter written against the wrong type is a mistake in the query rather than a fact about the file.

The bounds do not have to be a whole row group's. PageBounds is the same shape for one page, and the same arithmetic decides it.

type RLEDecoder added in v0.0.8 

type RLEDecoder struct {
	// contains filtered or unexported fields
}

RLEDecoder reads values written in the encoding parquet calls RLE, which is the hybrid of repeated runs and bit packed runs.

The width is how many bits each value takes and is not in the data, so it has to be worked out from the schema or read from the front of the page by whoever is calling. A width of nought is a column that only has one possible value, which happens to every level of a required column, and it is a real width rather than an error: the runs are still counted, so the decoder still knows how many values there are.

The zero value is a decoder of no values. Use NewRLEDecoder or Reset.

func NewRLEDecoder added in v0.0.8 

func NewRLEDecoder(data []byte, width int) (*RLEDecoder, error)

NewRLEDecoder returns a decoder reading width bit values out of data.

The data is the run of bytes and nothing around it. Neither of the two things the format puts in front of these bytes is read here: a data page written the first way puts four bytes of length in front of each of its two level runs, and dictionary indices have their width in the byte in front of them, and both of those belong to whoever is taking the page apart.

func (*RLEDecoder) Read added in v0.0.8 

func (d *RLEDecoder) Read(dst []int32) (int, error)

Read decodes values into dst and returns how many it wrote. It returns io.EOF once the data has been read to the end, in the way an io.Reader does: the call that reads the last values returns them with a nil error and the call after it returns io.EOF.

A run that the data does not hold is an error, and values decoded before the bad run are returned along with it, since a caller that wanted the levels of a page has nothing to do with a prefix of them but a caller counting what it got is entitled to see the count.

func (*RLEDecoder) Reset added in v0.0.8 

func (d *RLEDecoder) Reset(data []byte, width int) error

Reset points the decoder at other bytes, so that a scan reading a thousand pages of one column does not allocate a decoder for each of them.

type RLEEncoder added in v0.0.17 

type RLEEncoder struct {
	// contains filtered or unexported fields
}

RLEEncoder writes values in the encoding parquet calls RLE, which is the hybrid of repeated runs and bit packed runs.

The width is how many bits each value takes and it does not go into the data, the same way the decoder does not read it from there. Levels get it from the schema and dictionary indices get it from a byte the page writer puts in front of these bytes, so both of those belong to whoever is putting the page together.

A width of nought is a column with one possible value, which is what the levels of a required column would be and what the indices of a dictionary of one value are. It is a real width and not an error: the runs are still counted, so the values are still there to be counted back out, and every one of them has to be nought because nought is all that fits.

The zero value writes no values. Use NewRLEEncoder or Reset.

func NewRLEEncoder added in v0.0.17 

func NewRLEEncoder(width int) (*RLEEncoder, error)

NewRLEEncoder returns an encoder writing width bit values.

func (*RLEEncoder) Finish added in v0.0.17 

func (e *RLEEncoder) Finish() []byte

Finish closes the run being built and returns everything written.

The bytes are the encoder's own and are good until the next write to it. The last group is padded out to eight with noughts when it has to be, which is what every writer of this format does: a page says how many values it holds and a reader stops when it has them, so the values past the end are read and thrown away rather than mistaken for anything.

func (*RLEEncoder) Len added in v0.0.17 

func (e *RLEEncoder) Len() int

Len is how many bytes have been written down.

The values of the run being built are not among them, which is at most a group of eight and the count of a repeat, so this is what a page writer watching for its page to fill looks at and is a few bytes short of what the page will be.

func (*RLEEncoder) Reset added in v0.0.17 

func (e *RLEEncoder) Reset(width int) error

Reset empties the encoder and sets the width, keeping the buffer so that a column of a thousand pages allocates for the largest of them rather than for each of them.

func (*RLEEncoder) Write added in v0.0.17 

func (e *RLEEncoder) Write(vals []int32) error

Write adds values to the run being built.

A value wider than the width is an error and nothing of the call is written, since a value that does not fit would be read back as a different value and a page of levels that lost one is a page of nulls in the wrong places.

type Repetition 

type Repetition int32

Repetition says whether a column may be missing and whether it may repeat. It is how parquet writes down nesting and nullability at once: an optional field is a nullable column, and a repeated one is a list. 

const (
	NoRepetition Repetition = iota - 1
	Required
	Optional
	Repeated
)

The repetition types.

func (Repetition) String 

func (r Repetition) String() string

String returns the name the format gives the repetition, lowercased.

type RowGroup 

type RowGroup struct {
	Columns []ColumnChunk

	// TotalByteSize is the size of the values before compression, and
	// TotalCompressedSize is the number of bytes on disk.
	TotalByteSize       int64
	TotalCompressedSize int64

	NumRows int64

	// FileOffset is where the group starts, or zero for the files that leave
	// it out. The offsets on the column chunks are the ones to trust.
	FileOffset int64

	// Ordinal is which group this is, counting from zero.
	Ordinal int16
}

RowGroup is one horizontal slice of a file.

Every column has a chunk in every row group, so a row group is a set of rows that can be read without touching the rest of the file. This is the unit a scan skips: a filter that no row of a group can satisfy skips the group and never reads a page of it.

type SchemaElement 

type SchemaElement struct {
	Name string

	// Type is how the values are written down, or NoType for a group.
	Type Type

	// TypeLength is how many bytes a value takes, for FixedLenByteArray and
	// for nothing else.
	TypeLength int32

	// Repetition says whether the field may be missing or may repeat.
	Repetition Repetition

	// NumChildren is how many of the nodes that follow belong to this one.
	NumChildren int32

	// Converted is what the physical type means, in the older of the two ways
	// parquet says so, or NoConverted when the file did not say.
	Converted ConvertedType

	// Logical is the same thing in the newer way, which carries the parameters
	// the converted type could not. Its kind is NoLogical when the file did
	// not say.
	Logical LogicalType

	// Scale and Precision belong to a decimal and are here rather than on the
	// logical type because a file that only writes converted types has nowhere
	// else to put them.
	Scale     int32
	Precision int32

	// FieldID is the identifier a schema language gave the field, or zero when
	// the file did not say. Nothing in parquet uses it.
	FieldID int32
}

SchemaElement is one node of the schema tree.

The tree is written flat: this node, then its children, then their children. A node with children is a group and has no physical type, and a node with none is a column and has one. The root is the only node with no repetition.

type SizeStatistics added in v0.0.18 

type SizeStatistics struct {
	UnencodedBytes int64

	// HasUnencodedBytes says the writer wrote UnencodedBytes, which tells a
	// chunk that costs nothing to hold from one that said nothing. A column of
	// empty strings is the first and most files are the second.
	HasUnencodedBytes bool

	RepetitionHistogram []int64
	DefinitionHistogram []int64
}

SizeStatistics is what a column chunk costs once it is decoded.

A reader sizing a buffer knows the compressed and the uncompressed size of a chunk from the metadata, but uncompressed is still encoded, and a column of byte arrays written as a dictionary or as deltas is many times larger laid out than it is on disk. UnencodedBytes is that size, so a reader can allocate once rather than grow into it.

The two histograms are for repeated columns. Each counts the values of the chunk at every level from nought to the maximum, so a reader can work out how many rows a chunk holds, and how many of them are null or empty lists, without reading a level out of a page. They are absent for a flat column, where the answer is the number of values and the null count that are already there.

type Statistics 

type Statistics struct {
	MinValue []byte
	MaxValue []byte

	// MinExact and MaxExact say the bounds are values that are in the chunk
	// rather than bounds around them. A writer that truncates a long string to
	// keep the footer small says so this way.
	MinExact bool
	MaxExact bool

	Min []byte
	Max []byte

	// NullCount is how many values of the chunk are missing, and
	// DistinctCount is how many different ones there are. Both are only
	// meaningful when the flag next to them is set, since a file that says
	// nothing and a file that says zero are different files.
	NullCount     int64
	HasNullCount  bool
	DistinctCount int64
	HasDistinct   bool
}

Statistics is what a writer said about the values of a column chunk.

The values are the raw bytes of the physical type, so an Int32 column's smallest value is four bytes and a ByteArray column's is the string itself. Reading them means knowing the type, which is why they are not decoded here.

MinValue and MaxValue are the ones to use. Min and Max are the same idea from before the format pinned down how values are ordered, and are only safe to read for the types whose ordering nobody ever disagreed about. ReadBounds is what applies that rule and hands back the values decoded.

A bound the file did not write is nil and one it wrote empty is not, which is how a column of strings whose smallest value is the empty string is told from a column that said nothing about itself.

type TimeUnit 

type TimeUnit int32

TimeUnit is how fine a time or a timestamp is. There is no second unit, so the coarsest a parquet timestamp gets is milliseconds. 

const (
	NoUnit TimeUnit = iota
	Millis
	Micros
	Nanos
)

The time units.

func (TimeUnit) String 

func (u TimeUnit) String() string

String returns the name of the unit, lowercased.

type Type 

type Type int32

Type is the physical type of a column, which is how the values are written down rather than what they mean. A date and a signed integer are both Int32 and it takes the logical type to tell them apart. 

const (
	NoType Type = iota - 1
	Boolean
	Int32
	Int64
	Int96
	Float
	Double
	ByteArray
	FixedLenByteArray
)

The physical types.

func (Type) String 

func (t Type) String() string

String returns the name the format gives the type, lowercased.

type WriteOptions added in v0.0.20 

type WriteOptions struct {
	// RowGroupSize is how many rows go in a row group, and defaults to a
	// million.
	//
	// A row group is what a reader skips whole, so this is the granularity of
	// every filter that reads statistics. It is also what a reader has to hold
	// to read one column of one group, so a group of ten million rows of
	// strings is a large allocation on the other side of the file.
	RowGroupSize int

	// PageSize is roughly how many bytes of values go in a data page, and
	// defaults to a megabyte.
	//
	// It is roughly because a page holds a whole number of values and the last
	// one is what takes it over. A column of fixed width values divides, and a
	// column of byte arrays is added up as it goes.
	PageSize int

	// CreatedBy is the writer's name and version, which goes in the footer as
	// free text. It defaults to naming this library.
	CreatedBy string

	// Plain writes every value as it is and puts no dictionary pages in the
	// file.
	//
	// The default is to write a dictionary for each chunk of the columns that
	// have few enough distinct values for one, which is what every other
	// writer does and what makes a parquet file smaller than the values in it.
	// Turning it off gives a larger file that costs less to write, since what
	// a dictionary costs on the way out is a hash of every value.
	Plain bool
}

WriteOptions says how the file is laid out. The zero value, which is what a nil pointer means as well, writes the layout described on each field.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.