array

package
v0.0.27 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package array holds the values of one column.

An Array is a dtype, a length, a validity bitmap and the values themselves. It is one struct for every type rather than one type per dtype, because the engine dispatches on a dtype at runtime and a hierarchy of concrete types would mean an interface call per element or a type switch per batch. The dtype says which of the value fields is the one in use, and the constructors are what make sure it is.

An Array is immutable. Nothing here modifies one after it is built, which is what lets the executor hand the same chunk to several goroutines without copying it or locking it. Builders are how values get in, and Slice hands back a new Array that shares the same memory.

Nulls

A null is a clear bit in the validity bitmap. It is not a NaN and it is not a sentinel value, so a column of integers with a missing value is still a column of integers. NullCount is kept up to date rather than recomputed, because the branch that matters in every kernel is the one that asks whether there are any nulls at all, and a column with none is the common case.

A nil validity bitmap means no nulls. It is not the same as a bitmap with every bit set, in that it costs nothing and reads faster, and it is what a column that came from a file with no missing values gets.

Slicing

Slice is constant time. An Array carries an offset into the buffers it shares with the array it was sliced from, so slicing a chunk of a million rows into a morsel of eight thousand copies nothing. The one thing it has to do is count the nulls in the new range, which is a popcount over a few hundred bytes, because the whole point of NullCount is that reading it is free.

This is the layer the bitmap package's doc comment refers to when it says the layers above keep their own offset.

Building

A Builder is how values get in. It is for one dtype, decided when it is made, and it hands its memory over to the Array rather than copying it. A column with no nulls never allocates a validity bitmap, since the builder counts the values until the first null arrives and only fills in the bits before it if one ever does.

The constructors are the other way in, for a reader that already has the bytes laid out the way the column wants them, such as one reading a mapped file.

Dictionary encoding

A dictionary encoded column is small integers into a shared array of values, which is what pandas calls a Categorical and what most of the data in a Parquet file already is. NewDictionary builds one out of two ordinary arrays and the column keeps them both: Indices reads the integers as an array of the index type and Dictionary hands back the values.

Slicing one is still constant time, since it slices the indices and the values are shared by everything sliced from the same column.

Chunks

A Chunked is a column held as several arrays rather than one, which is what Arrow calls a ChunkedArray. A file arrives in record batches, and joining them into one array would mean copying every value to gain nothing. A kernel works on one chunk at a time and a chunk is an ordinary Array, so nothing below this line has to know whether the column it came from was chunked.

What is not here yet

The nested types, meaning List, Struct and Map. They are coming and they do not change the shape of what is here.

Stability: tier 1, stable.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Array 

type Array struct {
	// contains filtered or unexported fields
}

Array is the values of one column, or of one chunk of one column.

The zero Array is not usable. Use one of the constructors.

func New 

func New(dt dtype.DataType, length int, values *buffer.Buffer, valid *bitmap.Bitmap) (*Array, error)

New returns a fixed width array of length values of type dt, holding the values in values and the nulls in valid.

The array takes over both, meaning the caller must not modify them afterwards. Nothing is copied, which is what lets a column be read straight out of a mapped file.

A nil valid says every value is present. It may also be longer than the array, since a bitmap sized in whole bytes usually is. A bitmap with every bit set is dropped rather than kept, because a column with no nulls reads faster without one and the two say the same thing.

The buffer has to be long enough for length values, and may be longer. For a Bool column the values are bits, so it needs one byte per eight values.

Example

ExampleNew builds a column the way a reader would, by filling a buffer and a validity bitmap and handing both over. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/bitmap"
	"github.com/tamnd/kuma/buffer"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	values := buffer.New(4 * 8)
	for i, v := range []int64{1, 0, 3, 4} {
		for k := range 8 {
			values.Bytes()[i*8+k] = byte(uint64(v) >> (8 * k))
		}
	}

	valid := bitmap.NewSet(4)
	valid.Set(1, false) // the second value is missing

	a, err := array.New(dtype.Int64, 4, values, valid)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(a.Len(), a.NullCount())
	for i := range a.Len() {
		if a.IsNull(i) {
			fmt.Println("null")
			continue
		}
		fmt.Println(a.Value[int64](i))
	}
}

Output:
4 1
1
null
3
4

func NewDictionary added in v0.0.4 

func NewDictionary(indices, dict *Array) (*Array, error)

NewDictionary returns a dictionary encoded array holding the values of dict at the positions in indices.

The result borrows both, so slicing it is still constant time and two columns read out of the same file share the one copy of the values. The nulls are the indices' nulls.

Every index is checked against the length of dict, since the alternative is a read out of range in whatever kernel touches the column next, a long way from whoever built it. An index in a null slot is not checked, because a producer is allowed to leave anything there and the value behind it is never read.

Example

ExampleNewDictionary shows a column stored as indices into a shared set of values, which is what a Parquet file mostly holds and what pandas calls a Categorical. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
)

func main() {
	regions, err := array.NewDictionary(
		array.Of[int32](1, 0, 0, 1, 2),
		array.OfStrings("north", "south", "east"),
	)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(regions.Len(), regions.Dictionary().Len())
	for i := range regions.Len() {
		fmt.Print(string(regions.Dictionary().Bytes(regions.Index(i))), " ")
	}
	fmt.Println()
}

Output:
5 3
south north north south east

func NewNull 

func NewNull(length int) *Array

NewNull returns an array of length values, all of them missing, of the type that has no values. It panics if length is negative.

It carries no bitmap. A Null column is one where the type itself says every value is missing, so there is nothing to record per value and nothing to allocate, however long the column is.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
)

func main() {
	a := array.NewNull(1000)

	fmt.Println(a)
	fmt.Println(a.IsValid(0), a.Validity() == nil)
}

Output:
array.Array{null, len 1000, nulls 1000, offset 0}
false true

func NewStrings 

func NewStrings(dt dtype.DataType, d *strview.Data, valid *bitmap.Bitmap) (*Array, error)

NewStrings returns a String or Binary array over d, with the nulls in valid. The length is the number of values in d.

LargeString and LargeBinary are not here. They are the 64 bit offset layout that arrives over Arrow IPC and they are converted at that boundary, which they can be without losing anything, since a view has no global offset to overflow. A value is found by a block number and an offset inside that block, so a column of any size is a column with more blocks, and the only thing that does not fit is a single value longer than two gigabytes.

The array takes over d and valid, the same way New does.

func Of 

func Of[T Numeric](values ...T) *Array

Of returns an array of the given values, with no nulls, of the dtype that matches T. It is for tests, examples and the odd literal column, not for loading data, which goes through a builder.

This and the two below build the struct rather than going through New. There is nothing for New to check that is not already decided here, and an error return that cannot happen is a branch nobody can test.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
)

func main() {
	a := array.Of[int64](10, 20, 30)

	fmt.Println(a)
	fmt.Println(a.Values[int64]())
}

Output:
array.Array{int64, len 3, nulls 0, offset 0}
[10 20 30]

func OfBools 

func OfBools(values ...bool) *Array

OfBools returns a Bool array of the given values, with no nulls.

func OfStrings 

func OfStrings(values ...string) *Array

OfStrings returns a String array of the given values, with no nulls.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
)

func main() {
	a := array.OfStrings("kuma", "bear", "a value that is too long to live inside its view")

	for i := range a.Len() {
		fmt.Printf("%d %s\n", len(a.Bytes(i)), a.Bytes(i))
	}
}

Output:
4 kuma
4 bear
48 a value that is too long to live inside its view

func (*Array) Bool 

func (a *Array) Bool(i int) bool

Bool returns value i of a Bool column. It panics if the column is not Bool or if i is out of range.

Bool is the one fixed width type whose values are bits rather than bytes, so it is not reachable through Values and gets a method of its own.

func (*Array) Bools 

func (a *Array) Bools() *bitmap.Bitmap

Bools returns the values of a Bool column as a bitmap over the shared buffer, where value i of this array is bit Offset()+i. It panics if the column is not Bool.

This is for a kernel that wants to work a word at a time, which is the reason booleans are packed in the first place. Reading one value is Bool.

func (*Array) Buffer 

func (a *Array) Buffer() *buffer.Buffer

Buffer returns the buffer holding the fixed width values, or nil for a String, Binary or Null column. It is shared the same way Validity is.

For a dictionary encoded column the fixed width values are the indices, and Indices is the way to read them, since it comes back as an array of the index type and this comes back as bytes.

func (*Array) Bytes 

func (a *Array) Bytes(i int) []byte

Bytes returns value i of a column whose values are bytes rather than numbers, meaning String, Binary, FixedSizeBinary, the decimals and the intervals. It panics if the column is something else or if i is out of range.

The result aliases the column and the caller must not modify it. Converting it to a string copies, which is why that is left to the caller rather than done here for every value on the way past.

func (*Array) Clone 

func (a *Array) Clone() *Array

Clone returns a copy that shares no memory with a, holding only the values in range so that slicing a chunk out of a column and cloning it does not carry the rest of the column along with it.

func (*Array) DType 

func (a *Array) DType() dtype.DataType

DType returns the type of the values.

func (*Array) Dictionary added in v0.0.4 

func (a *Array) Dictionary() *Array

Dictionary returns the values a dictionary encoded column indexes into, or nil for a column that is not dictionary encoded.

It is the array this column was built with rather than a copy, which is what makes it shared, so it has its own length and its own nulls and neither is this column's.

func (*Array) Index added in v0.0.4 

func (a *Array) Index(i int) int

Index returns where in the dictionary value i of a dictionary encoded column is, or -1 when the value is missing. It panics if the column is not dictionary encoded or if i is out of range.

A missing value is -1 rather than whatever the producer happened to leave in the index, which is the same convention kernel.Take reads as a null, so a caller that forgets to ask IsNull gets an index out of range rather than a value that was never there.

func (*Array) Indices added in v0.0.4 

func (a *Array) Indices() *Array

Indices returns the index values of a dictionary encoded column as an array of the index type, or nil for a column that is not dictionary encoded.

It shares this column's memory and carries this column's nulls and offset, so a slice of a dictionary column has the indices of that slice. Reading them is then the ordinary Values and Value, since what comes back is an integer column like any other.

func (*Array) IsNull 

func (a *Array) IsNull(i int) bool

IsNull reports whether value i is missing.

func (*Array) IsValid 

func (a *Array) IsValid(i int) bool

IsValid reports whether value i is present. It panics if i is out of range, matching the behavior of an ordinary slice index.

func (*Array) Len 

func (a *Array) Len() int

Len returns the number of values.

func (*Array) NullCount 

func (a *Array) NullCount() int

NullCount returns how many of the values are missing. Reading it is free, which is why every operation that produces an Array pays to keep it right.

func (*Array) Offset 

func (a *Array) Offset() int

Offset returns where in the shared buffers this array starts, in elements. It is zero for an array that was not sliced.

func (*Array) Slice 

func (a *Array) Slice(i, j int) *Array

Slice returns values i through j-1 as a new array sharing this one's memory. It panics if the range is out of bounds.

It is constant time except for counting the nulls in the new range, which is a popcount over one byte per eight values and only happens at all when there are nulls to count.

Example

ExampleArray_Slice shows the two things a slice does: it shares the memory it came from, and it recounts the nulls in its own range. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/bitmap"
	"github.com/tamnd/kuma/buffer"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	valid := bitmap.NewSet(6)
	valid.Set(0, false)
	valid.Set(4, false)

	a, err := array.New(dtype.Int64, 6, buffer.New(6*8), valid)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(a)
	fmt.Println(a.Slice(1, 4))
	fmt.Println(a.Slice(1, 4).Slice(2, 3))
}

Output:
array.Array{int64, len 6, nulls 2, offset 0}
array.Array{int64, len 3, nulls 0, offset 1}
array.Array{int64, len 1, nulls 0, offset 3}

func (*Array) String 

func (a *Array) String() string

String returns a description of the array for debugging. It does not print the values, since an array is as long as a column.

func (*Array) Strings 

func (a *Array) Strings() *strview.Data

Strings returns the values of a String or Binary column, or nil for anything else. It is shared the same way Validity is, so value i of this array is value Offset()+i of the result.

func (*Array) Validity 

func (a *Array) Validity() *bitmap.Bitmap

Validity returns the validity bitmap, or nil when there are no nulls.

It is the bitmap this array shares with whatever it was sliced from, so bit i of this array is bit Offset()+i of the result and its length is not this array's length. A kernel walking one array wants IsValid. A kernel combining two of them wants this, and has to line up the offsets itself.

func (*Array) Value 

func (a *Array) Value[T Numeric](i int) T

Value returns value i of a fixed width column. It panics if T is not the type the column stores or if i is out of range.

It does not report whether the value is present. A null still has bytes behind it, usually zero, and reading them as though they were a value is the mistake this whole layout exists to prevent, so ask IsValid first.

func (*Array) Values 

func (a *Array) Values[T Numeric]() []T

Values returns the values of a fixed width column as a Go slice, without copying them. It panics if T is not the type the column stores.

This is a method with its own type parameter, which Go 1.27 allows on a concrete type. Before that it had to be a function, and a function cannot be the thing a caller reaches for after checking a dtype, because array.Values[int64](a) reads as a conversion of a and a.Values[int64]() reads as a question about a.

The result aliases the column and the caller must not modify it. It is the bytes the file was read into, reinterpreted, so writing to it writes to every other array sliced from the same buffer.

T has to match how the column is stored, not what it means. A timestamp column is int64, a date32 column is int32, and both are read as those. What this refuses is a reinterpretation that changes the width or the meaning of the bits, since reading a float64 column as an int64 gives numbers that are not wrong so much as unrelated.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
)

func main() {
	a := array.Of[float64](1.5, 2.5, 3.5, 4.5)

	sum := 0.0
	for _, v := range a.Slice(1, 3).Values[float64]() {
		sum += v
	}
	fmt.Println(sum)
}

Output:
6

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder accumulates values into an Array.

A builder is for one dtype, decided when it is made, and the append methods panic if they are handed something else. That is the same rule the read side follows: a column knows what it holds, and code that does not know is code with a bug in it rather than code that should be handed a conversion.

A column with no nulls never allocates a validity bitmap. The builder counts values until the first null arrives and only then fills in the bits for everything before it, so the common case of a column read from a file with nothing missing costs nothing to track.

The zero Builder is not usable. Use NewBuilder.

func NewBuilder 

func NewBuilder(dt dtype.DataType) (*Builder, error)

NewBuilder returns a builder for a column of type dt.

Example

ExampleNewBuilder is the way a reader builds a column: make room for the chunk, append values and nulls in the order they arrive, and finish. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	b, err := array.NewBuilder(dtype.Int64)
	if err != nil {
		fmt.Println(err)
		return
	}

	b.Grow(4)
	b.AppendValues([]int64{10, 20})
	b.AppendNull()
	b.Append[int64](40)

	a := b.Finish()
	fmt.Println(a)
	for i := range a.Len() {
		if a.IsNull(i) {
			fmt.Println("null")
			continue
		}
		fmt.Println(a.Value[int64](i))
	}
}

Output:
array.Array{int64, len 4, nulls 1, offset 0}
10
20
null
40

func (*Builder) Append 

func (b *Builder) Append[T Numeric](v T)

Append adds one value. It panics if T is not the type the column stores.

The type is checked against the layout of the column rather than its dtype, the same way Values is, so a timestamp column takes an int64 and a date32 column takes an int32.

func (*Builder) AppendBool 

func (b *Builder) AppendBool(v bool)

AppendBool adds one value to a Bool column. It panics if the column is not Bool.

func (*Builder) AppendBools 

func (b *Builder) AppendBools(vs []bool)

AppendBools adds every value in vs to a Bool column. It panics if the column is not Bool.

func (*Builder) AppendBytes 

func (b *Builder) AppendBytes(p []byte)

AppendBytes adds one value to a String, Binary or FixedSizeBinary column, or to one of the decimals or intervals. It copies p, so the caller may reuse it, which is what makes it safe to build a column out of a reader's own read buffer.

It panics if the column is something else, or if the column is fixed width and p is not exactly that wide.

func (*Builder) AppendNull 

func (b *Builder) AppendNull()

AppendNull adds one missing value.

A null still takes up its place in the values, since everything downstream indexes the values and the bitmap with the same number. What goes there is zero, and reading it as though it were a value is the mistake the bitmap exists to prevent.

func (*Builder) AppendNulls 

func (b *Builder) AppendNulls(n int)

AppendNulls adds n missing values. It panics if n is negative.

func (*Builder) AppendString 

func (b *Builder) AppendString(s string)

AppendString adds one value to a String or Binary column. It panics if the column is something else.

func (*Builder) AppendValues 

func (b *Builder) AppendValues[T Numeric](vs []T)

AppendValues adds every value in vs. It panics if T is not the type the column stores.

This is the one to reach for in a loop over a batch. The type is checked once instead of once per value, and the values go into the buffer as one copy.

func (*Builder) DType 

func (b *Builder) DType() dtype.DataType

DType returns the type of the column being built.

func (*Builder) Finish 

func (b *Builder) Finish() *Array

Finish returns the values appended so far as an Array and resets the builder.

The array takes the builder's memory rather than a copy of it, except for a Bool column, whose bits are copied once into an aligned buffer. Finish is what makes that safe: the builder comes back empty, so there is no way to write through it into a column that has already been handed out.

Example

ExampleBuilder_Finish shows what makes a builder worth reusing: it comes back empty, so the next column starts from nothing rather than from memory the last one is still reading. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	b, err := array.NewBuilder(dtype.String)
	if err != nil {
		fmt.Println(err)
		return
	}

	for _, chunk := range [][]string{{"kuma", "bear"}, {"one", "two", "three"}} {
		for _, s := range chunk {
			b.AppendString(s)
		}
		fmt.Println(b.Finish())
	}
}

Output:
array.Array{string, len 2, nulls 0, offset 0}
array.Array{string, len 3, nulls 0, offset 0}

func (*Builder) Grow 

func (b *Builder) Grow(n int)

Grow makes room for n more values. It panics if n is negative.

It is worth calling when the length is known, which for a reader it usually is, since a chunk is read into a buffer of a size the reader chose.

func (*Builder) Len 

func (b *Builder) Len() int

Len returns how many values have been appended, nulls included.

func (*Builder) NullCount 

func (b *Builder) NullCount() int

NullCount returns how many of them are missing.

func (*Builder) Reset 

func (b *Builder) Reset()

Reset drops everything and leaves a builder for the same dtype ready to use again.

It gives up the memory rather than keeping it, because the memory may have been handed to an Array by Finish and writing into it again would change a column somebody else is reading.

type Chunked 

type Chunked struct {
	// contains filtered or unexported fields
}

Chunked is a column held as a sequence of arrays rather than as one.

This is what Arrow calls a ChunkedArray. The name is shorter here because the package is already called array, and array.Chunked is what the thing is.

A column is chunked for two reasons. A file arrives in record batches and joining them into one array would mean copying every value to gain nothing. And a column longer than one allocation can hold has to be more than one allocation, which for a string column is any column with more than two gigabytes of text in it.

The chunks may be of any length and need not be of the same length. What they must be is the same type, since a column is one type by definition.

A Chunked is immutable, the same way an Array is. Append returns a new column and Slice returns a new column, and neither touches the one it was called on, which is what lets the executor hand the same column to several goroutines.

The zero Chunked is not usable. Use NewChunked.

func NewChunked 

func NewChunked(dt dtype.DataType, chunks ...*Array) (*Chunked, error)

NewChunked returns a column of type dt holding the given chunks in order.

Every chunk has to be of type dt, compared with dtype.Equal rather than by Kind, since a timestamp in microseconds and a timestamp in nanoseconds are the same Kind and are not the same column.

A chunk with no values is dropped. It holds nothing, and keeping it would mean every lookup had to step over a chunk that can never be the answer.

The chunks are not copied. The column shares them, and since an Array is immutable there is nothing to share badly.

Example

ExampleNewChunked shows the shape a reader produces: one array per batch, all of them one column. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	c, err := array.NewChunked(dtype.Int64,
		array.Of[int64](1, 2, 3),
		array.Of[int64](4, 5),
	)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(c)
	fmt.Println(c.Len(), c.Value[int64](4))
}

Output:
array.Chunked{int64, len 5, nulls 0, chunks 2}
5 5

func (*Chunked) Append 

func (c *Chunked) Append(chunks ...*Array) (*Chunked, error)

Append returns a column with the given chunks added to the end. The column it is called on is unchanged.

This is how a reader builds a column: finish an array for each batch it reads, and append it. The chunks already in the column are shared with the new one rather than copied.

func (*Chunked) At added in v0.0.4 

func (c *Chunked) At(i int) (chunk *Array, index int)

At returns the chunk holding value i and where in that chunk it is. It panics if i is out of range.

It is for a caller that wants something an Array can answer and a Chunked cannot, such as the dictionary a dictionary encoded column points at, which each chunk carries its own of.

func (*Chunked) Bool 

func (c *Chunked) Bool(i int) bool

Bool returns value i of a Bool column. It panics if the column is not Bool or if i is out of range.

func (*Chunked) Bytes 

func (c *Chunked) Bytes(i int) []byte

Bytes returns value i of a column whose values are bytes rather than numbers. It panics if the column is something else or if i is out of range.

The result aliases the chunk it came from and the caller must not modify it.

func (*Chunked) Chunk 

func (c *Chunked) Chunk(i int) *Array

Chunk returns chunk i. It panics if i is out of range.

func (*Chunked) Chunks 

func (c *Chunked) Chunks() []*Array

Chunks returns the chunks in order.

The result shares the column's own slice and the caller must not modify it. This is the loop a kernel runs: read each chunk as a slice of values through Values and do the work there, rather than asking this column for one value at a time.

Example

ExampleChunked_Chunks is the loop a kernel runs. Each chunk is a plain Go slice, and the work happens there rather than one value at a time through the column. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	c, err := array.NewChunked(dtype.Float64,
		array.Of[float64](1.5, 2.5),
		array.Of[float64](3.5, 4.5, 5.5),
	)
	if err != nil {
		fmt.Println(err)
		return
	}

	sum := 0.0
	for _, chunk := range c.Chunks() {
		for _, v := range chunk.Values[float64]() {
			sum += v
		}
	}
	fmt.Println(sum)
}

Output:
17.5

func (*Chunked) DType 

func (c *Chunked) DType() dtype.DataType

DType returns the type of the column.

func (*Chunked) IsNull 

func (c *Chunked) IsNull(i int) bool

IsNull reports whether value i is missing. It panics if i is out of range.

func (*Chunked) IsValid 

func (c *Chunked) IsValid(i int) bool

IsValid reports whether value i is present. It panics if i is out of range.

func (*Chunked) Len 

func (c *Chunked) Len() int

Len returns how many values the column holds, across all of its chunks.

func (*Chunked) NullCount 

func (c *Chunked) NullCount() int

NullCount returns how many of them are missing.

func (*Chunked) NumChunks 

func (c *Chunked) NumChunks() int

NumChunks returns how many chunks the column is held in. It is not the same as the number of chunks NewChunked was handed, since the empty ones are dropped.

func (*Chunked) Slice 

func (c *Chunked) Slice(i, j int) *Chunked

Slice returns the values from i up to but not including j, as a column. It panics unless 0 <= i <= j <= Len.

The chunks the range covers whole are shared as they are, and the one or two at the ends are sliced, which is constant time each. So the cost is a binary search and a null count over the two partial chunks, whatever the length of the range in between.

Example

ExampleChunked_Slice shows what a slice of a chunked column costs. The chunks the range covers whole are the same arrays, shared rather than copied, and only the ones at the ends are cut. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
)

func main() {
	c, err := array.NewChunked(dtype.Int64,
		array.Of[int64](0, 1, 2),
		array.Of[int64](3, 4, 5),
		array.Of[int64](6, 7, 8),
	)
	if err != nil {
		fmt.Println(err)
		return
	}

	s := c.Slice(1, 8)
	fmt.Println(s)
	fmt.Println(s.Chunk(1) == c.Chunk(1))
	for _, chunk := range s.Chunks() {
		fmt.Println(chunk.Values[int64]())
	}
}

Output:
array.Chunked{int64, len 7, nulls 0, chunks 3}
true
[1 2]
[3 4 5]
[6 7]

func (*Chunked) String 

func (c *Chunked) String() string

String returns a short description of the column, for a log line or a test failure. It is not the values.

func (*Chunked) Value 

func (c *Chunked) Value[T Numeric](i int) T

Value returns value i. It panics if T is not the type the column stores or if i is out of range.

It is for asking about one value, and it costs a binary search over the chunks to find which one holds it. A kernel reads Chunks instead and works on each of them as a slice.

type Numeric 

type Numeric interface {
	int8 | int16 | int32 | int64 |
		uint8 | uint16 | uint32 | uint64 |
		float32 | float64
}

Numeric is every Go type a fixed width column can be read as.

The types are exact rather than approximate. A column holds machine numbers, and a named type whose underlying type is int64 is a question for the layer that knows what the name means, not for the layer that owns the bytes.

type Table added in v0.0.15 

type Table struct {
	// Schema is the name, type and nullability of each column, in order.
	Schema dtype.Schema

	// Columns holds one column per field of the schema, in the same order.
	Columns []*Chunked
}

Table is a schema and the columns that go with it, which is a frame with the frame taken off.

It is what a file reader returns and what a file writer takes. A caller who wants rows and names and a query engine wants the frame the root package builds out of one of these; a caller who wants the columns wants this.

It is here rather than in one of the format packages because every one of them means the same thing by it, and a table that came out of a CSV file and a table that came out of a parquet file should be the same type rather than two types a caller has to convert between.

func (*Table) NumCols added in v0.0.15 

func (t *Table) NumCols() int

NumCols returns how many columns the table has.

func (*Table) NumRows added in v0.0.15 

func (t *Table) NumRows() int

NumRows returns how many rows the table has.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.