kernel

package
v0.0.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 28, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package kernel is the compute layer, where a column turns into another column.

Everything here works on arrays rather than on frames. A kernel is handed the values, does one job over all of them, and hands back new values, so the layer above can decide what a row is and this layer can concentrate on doing the same thing a million times without stopping to ask.

The rules

SIMD is an optimization and never a dependency. Everything in this package has a plain Go implementation that is built into every binary, on every platform, with no build tag. A vectorized version, when it arrives, is checked against the plain one, and when the two disagree the plain one is right by definition.

No exported signature names a type from the simd packages. Kernels take and return arrays and slices. That is what keeps an experimental package that has already broken once from being something a caller of this library has to know about.

A kernel panics for a mistake about types and lengths, the same way indexing a slice panics, since handing a gather a column it cannot read is a bug in the program rather than something the data did. Nothing here returns an error yet, and if something does it will be for a condition the data can cause.

What is here

Take, which reads values out of a column at the positions given, and Filter, which keeps the values a boolean mask selects. Everything that reorders or drops rows goes through one of those two, so joins, sorts, limits and predicates all come back here in the end.

Cast, which turns the values of a column into another type, and SortIndex, which works out the order rows go in and leaves the moving to Take.

Compare, which is the six comparisons, Arith, which is the five arithmetic operators, and And, Or and Not, which are the three valued logic that a column with holes in it needs. These are the three that read two columns at once, so a column of one value on either side is that value against every row of the other, which is how a comparison against a literal is written without building a column of copies of it.

GroupBy, which divides rows up by the values of some key columns, and the aggregations that run over what it produces: Sum, Mean, Count, Size, Min, Max, First, Last, Var, Std, Median, Quantile and NUnique. An aggregation over a whole column is an aggregation over OneGroup, so there is one of each rather than two.

Join, which works out which rows of two tables go together, in all seven of the ways SQL can. Like a sort it returns positions rather than a table, so building the result is Take's job and a caller who only wants to know what matched does not pay to build one.

IsNull and IsNotNull, which turn what is missing into a boolean column, FillNull, which puts a value where nothing was, and KeepIndex, which is the positions of the rows that have enough of their values to be worth keeping. The first two are a copy of a bitmap and the last returns positions, so the only one of the three that writes a value per row is the fill.

These are the reference implementations and they are not the fast ones. They append a value at a time, which is the version that is obviously right when read next to the definition of what a gather is. The one that writes a run of values into a buffer in one go, and the vectorized one after that, are both checked against what is here.

Stability: tier 2, evolving.

Document 11 has this package at tier 3, on the grounds that it will one day be full of build tagged files calling an unstable package. The exported surface never names any of that, which is the whole point of the second rule above, so the churn stays inside the package and the tier says what a caller can rely on rather than what the implementation is made of.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var ErrDivideByZero = errors.New("division by zero")

ErrDivideByZero is what dividing by zero in an integer column gives.

Functions

func And 

func And(a, b *array.Chunked) (*array.Chunked, error)

And returns the logical and of two boolean columns.

The logic is three valued, because a column has a third thing a value can be. A missing value is not known rather than false, so false and null is false, since nothing the unknown value could turn out to be would make the pair true, while true and null is null, since it depends. That is Kleene's rule, it is what SQL does, and it is what makes a filter on two conditions agree with a filter on each of them in turn.

A column of one value on either side is that value against every row of the other, so a predicate can be turned off with a single false without building a column of them.

It returns an error unless both columns hold conditions, since the column is often one a caller picked out of a file at runtime.

func Arith 

func Arith(a, b *array.Chunked, op ArithOp) (*array.Chunked, error)

Arith returns a column holding, for each row, the value in a combined with the value in b under the operator op.

The result has the type the two columns have in common, which for arithmetic means the same type on both sides. An int64 column plus a float64 column is an error naming the cast to write, not a quiet upcast, for the reason dtype.Coerce gives. A column of one value on either side is that value against every row of the other, which is how adding a literal is written.

The answer is the one the Go operator gives on the same two values, which is the rule worth stating once because it settles three questions that other libraries answer differently. Integer arithmetic wraps rather than widening or failing. Integer division truncates toward zero, so 7 / 2 is 3 and not 3.5, and a caller who wants the fraction casts first. Float division by zero gives an infinity or a NaN, because that is what a float64 means.

Integer division and remainder by zero have no answer at all, so they are an error naming the row, in the same shape as the one a cast that does not fit gives.

A missing value on either side gives a missing value out, since there is nothing to add.

Not yet: the decimals, where the scale of the result is a decision rather than a lookup, and the temporal types, where a timestamp plus a duration has to reconcile two units.

func Cast 

func Cast(c *array.Chunked, to dtype.DataType) (*array.Chunked, error)

Cast returns a column holding the values of c in the type to.

A value that will not fit is an error and the whole cast fails. Casting an int64 column to int8 when one row holds 400, or a string column to float64 when one row holds "n/a", stops and says which row it was. That is the right default because the alternative silently changes data, and a column with one bad row in it is nearly always a mistake somewhere upstream rather than something to paper over.

TryCast is the same cast with that one decision reversed.

A cast between two types that mean nothing to each other is an error before any value is read, so a plan fails while it is being built rather than partway through the second file. dtype.CanCast is the same question asked on its own.

The conversions are the obvious ones. A number becomes another number, a boolean, or its decimal text. A boolean becomes zero or one, or "true" and "false". Text is parsed into a number or a boolean. Bytes become text once they are checked for being valid UTF-8, and text becomes bytes for free. A temporal column and a number of the same width convert into each other by reinterpreting the stored count, which is what turns a timestamp into the microseconds it is made of and back.

A float becomes an integer by throwing the fraction away, the way a Go conversion does. It is the value that has to fit, not the fraction, so 3.9 becomes 3 and 300.0 does not become an int8. NaN and the infinities fit nowhere and are always the error case.

Text is parsed as the destination and not as a number in general. Casting "3.9" to int64 fails rather than producing 3, because the caller who wrote int64 said what they expected the file to contain.

A null in becomes a null out. A cast to the null type throws every value away, which is allowed because it takes saying so.

Not yet: the calendar side of the temporal types, meaning a change of unit, a formatted date and a parsed one. The decimals and the intervals are not here either. Each of those is an error saying as much rather than a wrong answer.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	read, _ := array.NewChunked(dtype.String, array.OfStrings("189", "411", "190"))

	prices, err := kernel.Cast(read, dtype.Int32)
	if err != nil {
		fmt.Println(err)
		return
	}
	for i := range prices.Len() {
		fmt.Println(prices.Value[int32](i))
	}
}

Output:
189
411
190
Example (DoesNotFit)

A value with no answer in the new type stops the cast and says which row it was, which is what makes a bad file findable. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	read, _ := array.NewChunked(dtype.String, array.OfStrings("189", "n/a", "190"))

	_, err := kernel.Cast(read, dtype.Int32)
	fmt.Println(err)
}

Output:
kernel: cannot cast string to int32: row 1 is "n/a": invalid syntax

func Compare 

func Compare(a, b *array.Chunked, op CompareOp) (*array.Chunked, error)

Compare returns a boolean column saying, for each row, whether the value in a stands in the relation op to the value in b.

A column of one value on either side is that value against every row of the other, which is how a comparison against a literal is written. Two columns of different lengths that are not that panic.

A missing value compares to nothing, so a null on either side gives a null rather than a false. That is what SQL does and what Polars does, and it is what makes Filter drop the row: a row nobody can say belongs in the result does not go in it. The two columns have to have a type in common, which is dtype.Coerce's question, and an int64 column against a float64 column is an error there rather than a quiet upcast.

NaN is unordered, so every comparison against it is false except !=, which is true. That is the IEEE rule and it is what the Go operators do, so a comparison here gives the same answer as the same comparison written out over the values. Polars decided the other way and calls NaN equal to itself. The pandas answer is the IEEE one, by way of numpy.

Not yet: the decimals, the intervals and the nested types, which have no order here for the same reason SortIndex gives.

func Count 

func Count(c *array.Chunked, g *Groups) *array.Chunked

Count returns how many values each group has, not counting the missing ones. It is never itself missing, since a group with nothing in it has none of it.

func FillNull 

func FillNull(c *array.Chunked, fill *array.Array) (*array.Chunked, error)

FillNull returns a column with every missing value of c replaced by the one value in fill.

There is no per type code here, and that is deliberate. The column and the one value fill are chained into a single column, which costs nothing because a chunked column is a list of chunks, and then the answer is a gather that takes position i where there is a value and the fill's position where there is not. Take already knows how to read every type, so this works for strings and timestamps and decimals without any of them being named.

The result is a new column, since a fill has to write the values it changed and there is nowhere to write them but a new buffer. A column with nothing missing is handed straight back, because there is nothing to change and nothing to copy.

It returns an error if fill is not exactly one value, if it is a different type from c, or if the value in it is itself missing, all of which are the caller asking for something that has no meaning rather than a bug in the program.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	b, err := array.NewBuilder(dtype.Int64)
	if err != nil {
		panic(err)
	}
	b.Append(int64(1))
	b.AppendNull()
	b.Append(int64(3))

	qty, err := array.NewChunked(dtype.Int64, b.Finish())
	if err != nil {
		panic(err)
	}

	filled, err := kernel.FillNull(qty, array.Of(int64(0)))
	if err != nil {
		panic(err)
	}
	for i := range filled.Len() {
		fmt.Println(filled.Value[int64](i))
	}
}

Output:
1
0
3

func Filter 

func Filter(c, mask *array.Chunked) *array.Chunked

Filter returns a column holding the values of c that mask selects, in the order they were in.

A null in the mask selects nothing. It is not a value that happens to be false, it is the absence of an answer, and a row nobody can say belongs in the result does not go in the result. That is what Polars does. The pandas answer was a warning and is now an error, which is the same behavior with more noise around it.

A filter is a gather, so a dictionary encoded column comes back dictionary encoded and pointing at the same values it went in with. See Take.

It panics if the two columns are not the same length, or if mask is not a boolean column.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	prices, _ := array.NewChunked(dtype.Float64, array.Of(189.5, 411.2, 190.1))
	mask, _ := array.NewChunked(dtype.Bool, array.OfBools(true, false, true))

	got := kernel.Filter(prices, mask)
	for i := range got.Len() {
		fmt.Println(got.Value[float64](i))
	}
}

Output:
189.5
190.1

func First 

func First(c *array.Chunked, g *Groups) *array.Chunked

First returns the first value of each group, skipping the missing ones, in the column's own type. A group whose values are all missing is missing.

This is pandas' first rather than SQL's ANY_VALUE. The row that is first whether or not its value is there is Groups.FirstRows, which a caller can gather at.

func Indices 

func Indices(mask *array.Chunked) []int

Indices returns the positions mask selects, in order.

This is the half of a filter that does not depend on the column being filtered, and it is exported because a frame filters every one of its columns with the same mask and should only answer this question once.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	mask, _ := array.NewChunked(dtype.Bool, array.OfBools(false, true, true, false, true))

	fmt.Println(kernel.Indices(mask))
}

Output:
[1 2 4]

func IsNotNull 

func IsNotNull(c *array.Chunked) *array.Chunked

IsNotNull returns a boolean column that is true where c has a value. It is IsNull the other way round, and it is the mask a caller wants far more often, since dropping the rows with nothing in them is what most callers are really asking for.

It panics if c is nil.

func IsNull 

func IsNull(c *array.Chunked) *array.Chunked

IsNull returns a boolean column that is true where c has no value.

The result has no nulls of its own. Whether a value is missing is always known, even when the value is not, so the answer is a plain boolean and not a boolean that might itself be missing. That is the difference between this and a comparison, where pandas gives back NaN for a row it could not compare.

It panics if c is nil.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	b, err := array.NewBuilder(dtype.Int64)
	if err != nil {
		panic(err)
	}
	b.Append(int64(1))
	b.AppendNull()
	b.Append(int64(3))

	qty, err := array.NewChunked(dtype.Int64, b.Finish())
	if err != nil {
		panic(err)
	}

	missing := kernel.IsNull(qty)
	for i := range missing.Len() {
		fmt.Println(missing.Bool(i))
	}
}

Output:
false
true
false

func KeepIndex 

func KeepIndex(cols []*array.Chunked, rows, present int) []int

KeepIndex returns the positions of the rows where at least present of cols have a value, in order. It is what dropping the rows that are too empty to be worth keeping comes down to, and the positions go straight into Take.

Every column has to be rows long. A present of zero or less keeps every row, since every row has at least nothing, and a present larger than the number of columns keeps none of them.

A column with nothing missing is counted without being read, so a frame of complete data is answered by counting the columns rather than the rows. The columns that can fail are read through their validity bitmaps, so a row costs a shift and a mask rather than the binary search that finding one value in a chunked column costs.

Example

KeepIndex answers over several columns at once, which is what makes it one call rather than a mask per column and then an and. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	b, err := array.NewBuilder(dtype.Int64)
	if err != nil {
		panic(err)
	}
	b.Append(int64(1))
	b.AppendNull()
	b.Append(int64(3))
	qty, err := array.NewChunked(dtype.Int64, b.Finish())
	if err != nil {
		panic(err)
	}

	b.AppendNull()
	b.Append(int64(2))
	b.Append(int64(3))
	fee, err := array.NewChunked(dtype.Int64, b.Finish())
	if err != nil {
		panic(err)
	}

	cols := []*array.Chunked{qty, fee}
	fmt.Println(kernel.KeepIndex(cols, 3, 2))
	fmt.Println(kernel.KeepIndex(cols, 3, 1))
}

Output:
[2]
[0 1 2]

func Last 

func Last(c *array.Chunked, g *Groups) *array.Chunked

Last returns the last value of each group, skipping the missing ones. It is First read backwards.

func Max 

func Max(c *array.Chunked, g *Groups) (*array.Chunked, error)

Max returns the largest value of each group, in the column's own type. It is Min the other way up, with NaN winning over every number.

func Mean 

func Mean(c *array.Chunked, g *Groups) (*array.Chunked, error)

Mean returns the average of each group, as a float64.

It is the total divided by how many values there were, not by how many rows there were, so a column with holes in it averages the values it has. A group with no values at all is missing rather than zero, since the average of nothing is not a number and saying zero would be inventing one.

Booleans average as ones and zeros, so the result is the fraction that are true.

It reports an error for the same columns Sum does, and for durations, since the average of two spans is a span and this returns a float64.

func Median 

func Median(c *array.Chunked, g *Groups) (*array.Chunked, error)

Median returns the middle value of each group, as a float64.

It is Quantile at a half with Linear interpolation, which is the pandas default and means an even number of values averages the two in the middle.

func Min 

func Min(c *array.Chunked, g *Groups) (*array.Chunked, error)

Min returns the smallest value of each group, in the column's own type.

The order is the one SortIndex uses, so strings compare by their bytes and NaN is larger than every number. A group with no values is missing.

It reports an error for a column there is no order for, which today means the decimals and the nested types.

func NUnique 

func NUnique(c *array.Chunked, g *Groups) (*array.Chunked, error)

NUnique returns how many distinct values each group has, not counting the missing ones. It is what pandas calls nunique and SQL calls COUNT DISTINCT.

Distinct means the same thing it means to GroupBy, since it is the same encoding doing the deciding, so all the NaNs count as one value and negative zero counts as zero.

It reports an error for the columns GroupBy refuses, which today means the nested types.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	visitor, err := array.NewChunked(dtype.String,
		array.OfStrings("ana", "bo", "ana", "cy", "bo", "ana"))
	if err != nil {
		panic(err)
	}
	day, err := array.NewChunked(dtype.Int32, array.Of[int32](1, 1, 1, 2, 2, 2))
	if err != nil {
		panic(err)
	}

	g, err := kernel.GroupBy(day)
	if err != nil {
		panic(err)
	}
	seen, err := kernel.NUnique(visitor, g)
	if err != nil {
		panic(err)
	}
	n := kernel.Count(visitor, g)

	for i := range g.NumGroups() {
		fmt.Println(g.Keys()[0].Value[int32](i), n.Value[int64](i), seen.Value[int64](i))
	}
}

Output:
1 3 2
2 3 3

func Not 

func Not(c *array.Chunked) (*array.Chunked, error)

Not returns the negation of a boolean column. A missing value stays missing, since the negation of a thing nobody knows is another thing nobody knows.

It returns an error unless the column holds conditions.

func Or 

func Or(a, b *array.Chunked) (*array.Chunked, error)

Or returns the logical or of two boolean columns, under the same three valued rule that And describes. True or null is true, false or null is null.

func Quantile 

func Quantile(c *array.Chunked, g *Groups, q float64, how Interpolation) (*array.Chunked, error)

Quantile returns the value at position q of each group, as a float64, where q runs from zero for the smallest to one for the largest.

The values of a group are put in order and then read at q of the way along, which for a q that does not land on a value is decided by how. A group with no values is missing.

NaN is a value and sorts after every number, the same as everywhere else here, so a group with a NaN in it has one at the top end and a quantile near one will find it. That is the answer that says the computation went wrong rather than the one that hides it.

It reports an error if q is outside zero to one, if how is not one of the five, or if the column is not numeric.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	latency, err := array.NewChunked(dtype.Float64,
		array.Of(12.0, 19.0, 15.0, 240.0, 14.0, 17.0, 13.0, 16.0, 18.0, 11.0))
	if err != nil {
		panic(err)
	}

	whole := kernel.OneGroup(latency.Len())
	for _, q := range []float64{0.5, 0.9, 0.99} {
		got, err := kernel.Quantile(latency, whole, q, kernel.Linear)
		if err != nil {
			panic(err)
		}
		fmt.Printf("%.2f %.2f\n", q, got.Value[float64](0))
	}
}

Output:
0.50 15.50
0.90 41.10
0.99 220.11

func Size 

func Size(g *Groups) *array.Chunked

Size returns how many rows each group has, counting the rows whose value is missing. It is what pandas calls size next to count.

func SortIndex 

func SortIndex(keys ...Order) ([]int, error)

SortIndex returns the positions of the rows in sorted order, ready to hand to Take.

It returns positions rather than a sorted column because a sort of a table is one order applied to every column, and because the caller who wants the order itself, to apply to a second table or to look at, would otherwise have no way to ask for it.

The first key decides, and each later one breaks the ties of the one before. The sort is stable, so rows that every key calls equal come out in the order they went in.

Null placement is a property of the key and not of the direction. Asking for descending order does not move the nulls, which is what a database does with an explicit NULLS LAST and what makes a query that says both mean what it says. NaN is a value rather than a missing one, and it sorts after every number, so descending order puts it first.

It panics if there are no keys, if a key column is nil, or if two of them are different lengths, all of which are mistakes in the program rather than in the data. It returns an error for a column of a type there is no order for, since the key column is usually one a user picked at runtime.

Example

SortIndex works out the order and Take applies it, which is how a sort of a table is one order applied to every column. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	symbol, _ := array.NewChunked(dtype.String, array.OfStrings("NVDA", "AAPL", "MSFT"))

	idx, err := kernel.SortIndex(kernel.Order{Column: symbol})
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(idx)

	sorted := kernel.Take(symbol, idx)
	for i := range sorted.Len() {
		fmt.Println(string(sorted.Bytes(i)))
	}
}

Output:
[1 2 0]
AAPL
MSFT
NVDA
Example (SeveralKeys)

The first key decides and the later ones break its ties, and each key has its own direction. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	symbol, _ := array.NewChunked(dtype.String, array.OfStrings("b", "a", "b", "a"))
	qty, _ := array.NewChunked(dtype.Int64, array.Of[int64](2, 9, 7, 1))

	idx, err := kernel.SortIndex(
		kernel.Order{Column: symbol},
		kernel.Order{Column: qty, Descending: true},
	)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(idx)
}

Output:
[1 3 2 0]

func Std 

func Std(c *array.Chunked, g *Groups, ddof int) (*array.Chunked, error)

Std returns the standard deviation of each group, which is the square root of Var and takes the same ddof.

Example

A ddof of one is the sample standard deviation and a ddof of zero is the population one. The first divides by the number of values less one, which is the right thing when the values are a sample of something larger, and it is what pandas and Polars do when nobody says otherwise. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	c, err := array.NewChunked(dtype.Float64, array.Of(2.0, 4.0, 4.0, 4.0, 5.0, 5.0, 7.0, 9.0))
	if err != nil {
		panic(err)
	}

	g := kernel.OneGroup(c.Len())
	sample, err := kernel.Std(c, g, 1)
	if err != nil {
		panic(err)
	}
	population, err := kernel.Std(c, g, 0)
	if err != nil {
		panic(err)
	}

	fmt.Println(sample.Value[float64](0), population.Value[float64](0))
}

Output:
2.138089935299395 2

func Sum 

func Sum(c *array.Chunked, g *Groups) (*array.Chunked, error)

Sum returns the total of each group.

The result is wider than the input, so a column of int8 sums into int64 and a column of uint8 into uint64. That is what stops a total from overflowing at 127. An int64 column can still overflow, and it wraps rather than reporting an error, the same way Go's addition does everywhere else.

A group with nothing to add up sums to zero rather than to nothing, which is what pandas and Polars both answer and what the arithmetic says: zero is what you get by adding up no numbers. Mean of the same group is missing, because there is no such number.

Booleans sum as ones and zeros, so the total is how many are true.

It reports an error for a column there is no sensible total of, which is everything that is not a number, a boolean or a duration. Adding two dates together is not a date and adding two strings is not this operation.

func Take 

func Take(c *array.Chunked, idx []int) *array.Chunked

Take returns a column holding the values of c at the given positions, in the order given.

A position below zero produces a null. That is not a courtesy to sloppy callers, it is what a left join does with a row that matched nothing, and having one rule for it here means the join does not need a second pass to put the nulls in. A position at or past the length of the column panics, the same way indexing a slice does.

The result is a new column that shares nothing with c, since the values it wants are scattered through the old one and there is no way to point at them. That makes this the expensive operation it looks like: gathering a million rows out of a column writes a million values.

A dictionary encoded column is the exception and the reason the encoding is worth having. Only the indices are gathered and the result points at the same values as c, so taking a million rows out of a column of country codes writes a million int32s rather than a million strings.

It panics if c is nil or is a type this package cannot read yet, which today means the nested types.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	prices, _ := array.NewChunked(dtype.Float64, array.Of(189.5, 411.2, 190.1, 12.75))

	// The order a sort or a join worked out, somewhere else.
	got := kernel.Take(prices, []int{3, 0, 2})
	for i := range got.Len() {
		fmt.Println(got.Value[float64](i))
	}
}

Output:
12.75
189.5
190.1
Example (Unmatched)

A position below zero is a null in the result, which is how an outer join says that a row matched nothing. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	names, _ := array.NewChunked(dtype.String, array.OfStrings("AAPL", "MSFT"))

	got := kernel.Take(names, []int{1, -1, 0})
	for i := range got.Len() {
		if got.IsNull(i) {
			fmt.Println("null")
			continue
		}
		fmt.Println(string(got.Bytes(i)))
	}
}

Output:
MSFT
null
AAPL

func TryCast 

func TryCast(c *array.Chunked, to dtype.DataType) (*array.Chunked, error)

TryCast is Cast with a value that does not fit becoming a null.

This is the cast to reach for when the data is known to be dirty and the plan is to count the nulls afterwards, which is a real thing to want when the alternative is a file of a million rows failing on the one that says "N/A". It is Polars with strict off, and SQL's TRY_CAST.

It still fails on a pair of types that mean nothing to each other, since no per row answer would make that cast into a sensible one.

Example

TryCast is the same cast with the bad row becoming a null, which is what to reach for when the plan is to count them afterwards. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	read, _ := array.NewChunked(dtype.String, array.OfStrings("189", "n/a", "190"))

	prices, err := kernel.TryCast(read, dtype.Int32)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(prices.Len(), "values,", prices.NullCount(), "of them missing")
}

Output:
3 values, 1 of them missing

func Var 

func Var(c *array.Chunked, g *Groups, ddof int) (*array.Chunked, error)

Var returns the variance of each group, as a float64.

The divisor is the number of values less ddof. A ddof of one is the sample variance, which is what pandas and Polars both do when nobody says otherwise, and a ddof of zero is the population variance, which is what numpy does. The difference matters on small groups and disappears on large ones, and neither of them is right often enough to be the only one offered.

A group with fewer values than the divisor needs is missing rather than infinite, so with the usual ddof of one a group of a single value has no variance.

It reports an error for a column there is no variance of, which is everything that is not a number or a boolean.

Types

type ArithOp 

type ArithOp uint8

ArithOp is one of the five arithmetic operators. 

const (
	OpAdd ArithOp = iota // addition
	OpSub                // subtraction
	OpMul                // multiplication
	OpDiv                // division
	OpMod                // remainder
)

The operators.

func (ArithOp) String 

func (o ArithOp) String() string

String returns the operator as it is written in Go.

type CastError 

type CastError struct {
	Row   int            // which value, counted from the start of the column
	From  dtype.DataType // the type of the column
	To    dtype.DataType // the type asked for
	Value string         // the value that would not fit, printed
	Err   error          // what a parser said, or nil when nothing was parsed
}

CastError says which row of a cast failed and why.

The row is counted across the whole column rather than within a chunk, because chunk boundaries are an accident of how the data was read and nobody looking for the bad row in a file cares where they fell.

func (*CastError) Error 

func (e *CastError) Error() string

func (*CastError) Unwrap 

func (e *CastError) Unwrap() error

Unwrap returns the parse error, so that errors.Is finds strconv.ErrRange through a cast that failed on a number that was too big for its type.

type CompareOp 

type CompareOp uint8

CompareOp is one of the six comparisons. 

const (
	OpEq CompareOp = iota // equal
	OpNe                  // not equal
	OpLt                  // less than
	OpLe                  // less than or equal
	OpGt                  // greater than
	OpGe                  // greater than or equal
)

The comparisons.

func (CompareOp) String 

func (o CompareOp) String() string

String returns the operator as it is written in Go.

type Groups 

type Groups struct {
	// contains filtered or unexported fields
}

Groups is rows divided up by the values of one or more key columns.

It is what GroupBy works out and what every aggregation is handed. Working it out once and passing it around is the point: a query that asks for a sum, a mean and a count over the same keys divides the rows up once and then makes three cheap passes rather than three expensive ones.

The groups are numbered in the order they first appear in the rows, which is deterministic without being sorted. A caller who wants them sorted sorts the result, and pays for the sort only when they want it. The pandas default is to sort, and turning it off is the single most common thing people do to a group by there.

Example

A grouping is worked out once and handed to as many aggregations as the caller wants, which is the reason it is a value rather than an argument. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	day, err := array.NewChunked(dtype.Int32, array.Of[int32](1, 1, 2, 2, 2))
	if err != nil {
		panic(err)
	}
	price, err := array.NewChunked(dtype.Float64, array.Of(9.0, 11.0, 4.0, 6.0, 8.0))
	if err != nil {
		panic(err)
	}

	g, err := kernel.GroupBy(day)
	if err != nil {
		panic(err)
	}
	mean, err := kernel.Mean(price, g)
	if err != nil {
		panic(err)
	}
	high, err := kernel.Max(price, g)
	if err != nil {
		panic(err)
	}
	n := kernel.Count(price, g)

	for i := range g.NumGroups() {
		fmt.Println(g.Keys()[0].Value[int32](i), mean.Value[float64](i),
			high.Value[float64](i), n.Value[int64](i))
	}
}

Output:
1 10 11 2
2 6 8 3

func GroupBy 

func GroupBy(keys ...*array.Chunked) (*Groups, error)

GroupBy divides rows up by the values of the key columns.

Two rows are in the same group when every key agrees, and a missing value agrees with a missing value. That is what SQL does and what Polars does. The pandas default is to drop the rows whose key is missing, which loses data quietly and is the wrong thing for a library people will use to count things.

A dictionary encoded key groups by the value behind the index rather than by the index, so two chunks that hold the same country codes in a different order still group together, and a row pointing at a value the dictionary says is missing groups with the rows that have no value at all. The keys come back dictionary encoded and pointing at the values they came from.

NaN keys land in one group rather than in one group each. There are millions of bit patterns that are NaN and telling them apart would put a row in a group of its own for a reason nobody asked about. The same goes for negative zero, which groups with zero because it is equal to it.

It reports an error if a key column holds values that cannot be compared for equality, which today means the nested types.

It panics if there are no keys, if a key is nil, or if the keys are not all the same length, since all three are a mistake in the program rather than something the data did.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	symbol, err := array.NewChunked(dtype.String,
		array.OfStrings("NVDA", "AAPL", "NVDA", "AAPL", "NVDA"))
	if err != nil {
		panic(err)
	}
	qty, err := array.NewChunked(dtype.Int64, array.Of[int64](10, 3, 20, 4, 30))
	if err != nil {
		panic(err)
	}

	g, err := kernel.GroupBy(symbol)
	if err != nil {
		panic(err)
	}
	total, err := kernel.Sum(qty, g)
	if err != nil {
		panic(err)
	}

	for i := range g.NumGroups() {
		fmt.Println(string(g.Keys()[0].Bytes(i)), total.Value[int64](i))
	}
}

Output:
NVDA 60
AAPL 7

func OneGroup 

func OneGroup(n int) *Groups

OneGroup returns n rows in a single group.

This is what makes an aggregation over a whole column and an aggregation over a group the same piece of code. Sum of a series is Sum of one group, and there is no second implementation to keep in step with the first.

func (*Groups) FirstRows 

func (g *Groups) FirstRows() []int

FirstRows returns the first row of every group, in group order. The caller must not modify the result.

This is the positional first that SQL calls ANY_VALUE and pandas calls nth(0), and it is not what First returns, since that one skips over the missing values. Gathering a column at these positions is how to get it.

func (*Groups) IDs 

func (g *Groups) IDs() []int

IDs returns the group of every row, in row order. The caller must not modify the result.

func (*Groups) Keys 

func (g *Groups) Keys() []*array.Chunked

Keys returns the distinct key values, one row per group, in group order.

There is one column here for every column GroupBy was given, in the order they were given. The caller must not modify the result.

func (*Groups) Len 

func (g *Groups) Len() int

Len returns how many rows were grouped.

func (*Groups) NumGroups 

func (g *Groups) NumGroups() int

NumGroups returns how many distinct keys there are.

func (*Groups) Sizes 

func (g *Groups) Sizes() []int

Sizes returns how many rows are in every group, in group order, counting the rows whose values are missing. The caller must not modify the result.

type Interpolation 

type Interpolation int

Interpolation says what a quantile does when it lands between two values.

These are the five pandas has, under the same names numpy gives them, and they exist because there is no single right answer. A median of an even number of values is the obvious case: Linear and Midpoint average the two in the middle, Lower and Higher pick one of them, and Nearest picks whichever the position is closer to.

Example

The five interpolations differ only when the quantile falls between two values, which is most of the time on a small group. Lower and higher pick one of the neighbors, midpoint splits them evenly whatever the fraction was, and nearest picks the closer one. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	c, err := array.NewChunked(dtype.Float64, array.Of(1.0, 2.0, 3.0, 4.0))
	if err != nil {
		panic(err)
	}

	g := kernel.OneGroup(c.Len())
	for _, how := range []kernel.Interpolation{
		kernel.Linear, kernel.Lower, kernel.Higher, kernel.Nearest, kernel.Midpoint,
	} {
		got, err := kernel.Quantile(c, g, 0.25, how)
		if err != nil {
			panic(err)
		}
		fmt.Println(how, got.Value[float64](0))
	}
}

Output:
linear 1.75
lower 1
higher 2
nearest 2
midpoint 1.5

const (
	// Linear walks the fraction of the way from the lower value to the higher
	// one. It is what pandas and numpy do when nobody says otherwise.
	Linear Interpolation = iota

	// Lower takes the value below the position.
	Lower

	// Higher takes the value above the position.
	Higher

	// Nearest takes whichever of the two the position is closer to, and when it
	// is exactly between them takes the one at the even index, which is what
	// numpy's rounding does.
	Nearest

	// Midpoint takes the average of the two, whatever the position is between
	// them.
	Midpoint
)

The ways a quantile can land between two values.

func (Interpolation) String 

func (i Interpolation) String() string

String returns the pandas name of the interpolation.

type JoinType 

type JoinType int

JoinType is which rows of the two sides a join keeps. 

const (
	// InnerJoin keeps the pairs that matched and nothing else.
	InnerJoin JoinType = iota

	// LeftJoin keeps every left row, with the right side missing where nothing
	// matched. RightJoin is the same thing the other way round.
	LeftJoin
	RightJoin

	// OuterJoin keeps every row of both sides. It is what SQL calls a full
	// outer join.
	OuterJoin

	// SemiJoin keeps the left rows that matched, once each, and takes nothing
	// from the right side. It is the join for "which of these have one", and it
	// is an EXISTS in SQL.
	SemiJoin

	// AntiJoin keeps the left rows that matched nothing, which is a NOT EXISTS.
	AntiJoin

	// CrossJoin pairs every left row with every right row and looks at no keys
	// at all.
	CrossJoin
)

The seven joins. They are the seven SQL has, and they mean the same things here.

func (JoinType) String 

func (j JoinType) String() string

String returns the name of the join, as it would be written in a query.

type Order 

type Order struct {
	Column     *array.Chunked
	Descending bool
	NullsFirst bool
}

Order is one column of a sort, and how it takes part.

The zero value of the two flags is ascending with the nulls at the end, which is what pandas does when nobody says otherwise and what most databases do.

type Pairs 

type Pairs struct {
	// Left holds the left row of every output row, and Right the right row.
	// They are the same length except after a semi or an anti join, which take
	// nothing from the right side and leave Right nil.
	Left  []int
	Right []int
}

Pairs is which rows of the two sides a join put together.

A join returns positions rather than a joined table for the same reason a sort does: the result is one set of positions applied to every column of each side, and building the table is Take's job. It also means a caller who wants to know what matched, and not the table, does not pay to build one.

A position below zero means nothing matched, which is exactly what Take turns into a null, so an outer join needs no special handling anywhere downstream.

func Join 

func Join(left, right Side, how JoinType) (Pairs, error)

Join works out which rows of the two sides go together.

Rows match when they agree on every key, using the same encoding GroupBy uses, so an int8 and an int64 holding the same number match and every NaN matches every other NaN.

A missing key matches nothing, including another missing key. That is what SQL says and what Polars does, and it is the answer that keeps a join from gluing together every row whose field was left blank. It is not what pandas does, where merging on a column with NaN in it happily pairs the blanks up.

Output order is the left side's row order, with the matches of one left row in the right side's row order, so the result is deterministic and reads the way the input did. A right join is ordered by the right side. The rows an outer join adds for the unmatched right rows come at the end, in right side order, which is where SQL puts them too.

It panics if a side's row count disagrees with its key columns, if a key column is nil, or if the two sides have different numbers of keys, all of which are mistakes in the program rather than in the data. It returns an error for a key column of a type there is no encoding for, since that type usually comes from data, and for a keyed join with no keys or a cross join with some.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	trades, err := array.NewChunked(dtype.String,
		array.OfStrings("AAPL", "MSFT", "TSLA"))
	if err != nil {
		panic(err)
	}
	listed, err := array.NewChunked(dtype.String, array.OfStrings("MSFT", "AAPL"))
	if err != nil {
		panic(err)
	}
	sector, err := array.NewChunked(dtype.String, array.OfStrings("software", "hardware"))
	if err != nil {
		panic(err)
	}

	p, err := kernel.Join(
		kernel.Side{Rows: trades.Len(), Keys: []*array.Chunked{trades}},
		kernel.Side{Rows: listed.Len(), Keys: []*array.Chunked{listed}},
		kernel.LeftJoin)
	if err != nil {
		panic(err)
	}

	// A position below zero is a null, so the row that matched nothing needs no
	// handling of its own.
	symbols := kernel.Take(trades, p.Left)
	sectors := kernel.Take(sector, p.Right)
	for i := range p.Len() {
		if sectors.IsNull(i) {
			fmt.Println(string(symbols.Bytes(i)), "unknown")
			continue
		}
		fmt.Println(string(symbols.Bytes(i)), string(sectors.Bytes(i)))
	}
}

Output:
AAPL hardware
MSFT software
TSLA unknown
Example (Cross)

A cross join takes row counts rather than keys, because it looks at no values at all. It is the one join that turns two small tables into a large one, so it has to be asked for by name. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/kernel"
)

func main() {
	p, err := kernel.Join(kernel.Side{Rows: 2}, kernel.Side{Rows: 3}, kernel.CrossJoin)
	if err != nil {
		panic(err)
	}
	fmt.Println(p.Len(), p.Left, p.Right)
}

Output:
6 [0 0 0 1 1 1] [0 1 2 0 1 2]
Example (MissingKeys)

A missing key matches nothing, including another missing key, which is what SQL says. It is not what pandas does, where merging on a column with blanks in it pairs the blanks up with each other. 

package main

import (
	"fmt"

	"github.com/tamnd/kuma/array"
	"github.com/tamnd/kuma/dtype"
	"github.com/tamnd/kuma/kernel"
)

func main() {
	left, err := array.NewChunked(dtype.Int64, array.Of[int64](1, 2))
	if err != nil {
		panic(err)
	}

	b, err := array.NewBuilder(dtype.Int64)
	if err != nil {
		panic(err)
	}
	b.AppendNull()
	b.Append[int64](2)
	right, err := array.NewChunked(dtype.Int64, b.Finish())
	if err != nil {
		panic(err)
	}

	// The left side has no null in it, so this is the right side's null being
	// asked to match, and the only pair is the two twos.
	p, err := kernel.Join(
		kernel.Side{Rows: left.Len(), Keys: []*array.Chunked{left}},
		kernel.Side{Rows: right.Len(), Keys: []*array.Chunked{right}},
		kernel.InnerJoin)
	if err != nil {
		panic(err)
	}
	fmt.Println(p.Left, p.Right)
}

Output:
[1] [1]

func (Pairs) Len 

func (p Pairs) Len() int

Len returns how many rows the join produced.

type Side 

type Side struct {
	Rows int
	Keys []*array.Chunked
}

Side is one side of a join: how many rows it has and what to match on.

The row count is given rather than taken from the key columns because a cross join has no key columns and still has rows. For every other join the count has to agree with what the keys say, and it is checked.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.