dtype

package
v0.0.27 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 29, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Overview

Package dtype describes the types a column can hold.

A DataType says what the values in a column are and, for the parameterized types, what the parameters are. It says nothing about whether the values are present. Missing values are a validity bitmap on the array and a Nullable flag on the Field, deliberately not part of the type, because the alternative is pandas, where a column of integers turns into a column of floats the moment one value goes missing.

Types with no parameters are package level values: 

dtype.Int64
dtype.String

Types with parameters are ordinary structs, written as composite literals: 

dtype.Timestamp{Unit: dtype.Microsecond, Zone: "Europe/London"}
dtype.List{Elem: dtype.Int64}
dtype.Decimal128{Precision: 18, Scale: 2}

There are no constructor functions and nothing here panics. A literal can be written that does not describe a real type, such as a Time32 in nanoseconds or a decimal with a scale larger than its precision, and Validate is what reports that. The frame layer validates a schema once, when it is built, rather than every operation checking for itself.

Every type reports a Kind, which is the type with its parameters removed. Kind is what a kernel dispatch table keys on. It is not what two column types should be compared with, because a timestamp in microseconds and a timestamp in nanoseconds are the same Kind and are not interchangeable. Use Equal for that.

Stability: tier 1, stable.

Index

Examples

Constants

View Source 
const (
	// MaxDecimal128Precision is the largest precision a Decimal128 can hold.
	MaxDecimal128Precision = 38

	// MaxDecimal256Precision is the largest precision a Decimal256 can hold.
	MaxDecimal256Precision = 76
)

The limits on decimal precision, which come from how many digits fit in the underlying integer.

View Source 
const MaxNestingDepth = 64

MaxNestingDepth is how deep a type may be nested before Validate gives up.

The limit exists because DataType is an interface anyone can implement, so a type that contains itself is constructible, and a recursive walk over one never returns. Real schemas are a few levels deep. A list of structs of lists is three.

Variables

View Source 
var ErrTooDeep = errors.New("type nests too deeply")

ErrTooDeep is what Validate returns, wrapped, when a type nests past MaxNestingDepth.

Functions

func Bits 

func Bits(t DataType) (int, bool)

Bits returns the width of one value in bits and whether t has one. It is the function form of the FixedWidth interface, for the common case where the caller wants a number or a false rather than a type assertion.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	// Bool is one bit per value, not one byte, which anything sizing a buffer
	// has to know.
	for _, t := range []dtype.DataType{dtype.Bool, dtype.Int32, dtype.String} {
		bits, fixed := dtype.Bits(t)
		fmt.Println(t, bits, fixed)
	}
}

Output:
bool 1 true
int32 32 true
string 0 false

func CanCast 

func CanCast(from, to DataType) bool

CanCast reports whether an explicit cast from one type to another is a thing this library will attempt.

It is much looser than Coerce, and it has to be, because a cast is what the caller writes when Coerce has refused. Coerce answers "may I do this without being asked", and the answer is almost always no. CanCast answers "is this a meaningful thing to ask for", and the answer is usually yes.

It is a question about types, not about values. A permitted cast can still fail on a particular row: casting int64 to int8 overflows, casting string to int64 meets a row that is not a number, casting a nanosecond timestamp to seconds throws away the fraction. Whether such a row becomes an error or a null is the kernel's decision and the caller's option, and neither of those is a question this package can answer. What CanCast rules out is the cast that has no meaning at all, such as a list to a struct, so that the plan fails while it is being built rather than partway through the second file.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	// A cast is what the caller writes when Coerce has refused, so it allows
	// much more. Whether a particular row survives is decided when the values
	// are read.
	fmt.Println(dtype.CanCast(dtype.Int64, dtype.Float64))
	fmt.Println(dtype.CanCast(dtype.String, dtype.Timestamp{Unit: dtype.Microsecond}))

	// A duration is a span and a timestamp is a point, and turning one into the
	// other needs an origin that nobody has stated.
	fmt.Println(dtype.CanCast(dtype.Duration{Unit: dtype.Second}, dtype.Timestamp{Unit: dtype.Second}))
}

Output:
true
true
false

func Equal 

func Equal(a, b DataType) bool

Equal reports whether a and b are the same type, parameters included. Two nil types are equal and a nil is equal to nothing else.

A timestamp in microseconds and a timestamp in nanoseconds are not equal even though they are the same Kind, and a list of int64 is not equal to a list of int32. This is the comparison to use before deciding that two columns can be concatenated or joined.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	// Two timestamps of different resolutions are the same kind and not the
	// same type. Concatenating them without a cast would silently multiply
	// every value by a thousand.
	us := dtype.Timestamp{Unit: dtype.Microsecond}
	ns := dtype.Timestamp{Unit: dtype.Nanosecond}

	fmt.Println(us.Kind() == ns.Kind())
	fmt.Println(dtype.Equal(us, ns))
}

Output:
true
false

func IsBinary 

func IsBinary(t DataType) bool

IsBinary reports whether t holds opaque bytes, meaning Binary, LargeBinary or FixedSizeBinary.

func IsDecimal 

func IsDecimal(t DataType) bool

IsDecimal reports whether t is one of the fixed point decimal types.

func IsFloat 

func IsFloat(t DataType) bool

IsFloat reports whether t is one of the floating point types.

func IsInteger 

func IsInteger(t DataType) bool

IsInteger reports whether t is a signed or unsigned integer type. Bool is not an integer here, which is a deliberate difference from pandas, where a boolean column sums to an integer without anyone asking for it.

func IsNested 

func IsNested(t DataType) bool

IsNested reports whether t has child types, meaning the value of one row is itself made of other values.

func IsNumeric 

func IsNumeric(t DataType) bool

IsNumeric reports whether t is an integer, a float or a decimal.

func IsSigned 

func IsSigned(t DataType) bool

IsSigned reports whether t is one of the signed integer types.

func IsString 

func IsString(t DataType) bool

IsString reports whether t holds text, meaning String or LargeString.

func IsTemporal 

func IsTemporal(t DataType) bool

IsTemporal reports whether t is a date, a time, a timestamp, a duration or an interval.

func IsUnsigned 

func IsUnsigned(t DataType) bool

IsUnsigned reports whether t is one of the unsigned integer types.

func Validate 

func Validate(t DataType) error

Validate reports whether t describes a real type.

The parameterized types are composite literals, so a caller can write one that does not mean anything: a time32 in nanoseconds does not fit in 32 bits, a decimal with a scale larger than its precision has no digits left, a dictionary indexed by a string cannot be indexed. Validate is where those are caught, in one place, rather than in every kernel that receives a type.

It walks the children of the nested types, so validating the outermost type validates the whole tree, and the error says which part is wrong.

Timestamp.Zone is not checked. It is deliberately not resolved against the timezone database, because a binary built without tzdata, which is most containers, would then reject a schema that is entirely valid on the machine that wrote it. A zone is resolved where the arithmetic that needs it happens.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	// A time of day in nanoseconds does not fit in 32 bits.
	fmt.Println(dtype.Validate(dtype.Time32{Unit: dtype.Nanosecond}))

	// The error names the part of the tree that is wrong.
	fmt.Println(dtype.Validate(dtype.List{Elem: dtype.Decimal128{Precision: 0}}))

	fmt.Println(dtype.Validate(dtype.List{Elem: dtype.Int64}))
}

Output:
dtype: time32 unit must be s or ms, have ns
dtype: list: decimal128 precision 0 out of range 1 to 38
<nil>

Types

type DataType 

type DataType interface {
	// Kind returns the type with its parameters removed.
	Kind() Kind

	// String returns the canonical name of the type, parameters included. It
	// is unique: two types print the same if and only if Equal reports them
	// equal, with the one exception that metadata attached to the fields of a
	// struct is not printed. That is what makes the name usable in an error
	// message and in a test assertion.
	String() string
}

DataType is the type of the values in one column.

The interface is two methods because it needs runtime polymorphism and nothing else. Anything that varies by type, meaning width, coercion, casting and every kernel, is a function or a table outside the interface, so that adding one does not force every implementation to change. 

var (
	// Null is a column of nothing. Every value is missing and no storage is
	// allocated. It exists because a literal nil and an all missing column
	// read from a file both have to have a type, and because it is what makes
	// Coerce able to combine a missing value with anything.
	Null DataType = &variable{NullKind, "null"}

	// Bool is one bit per value, packed the same way a validity bitmap is.
	Bool DataType = &fixed{BoolKind, "bool", 1}

	Int8  DataType = &fixed{Int8Kind, "int8", 8}
	Int16 DataType = &fixed{Int16Kind, "int16", 16}
	Int32 DataType = &fixed{Int32Kind, "int32", 32}
	Int64 DataType = &fixed{Int64Kind, "int64", 64}

	Uint8  DataType = &fixed{Uint8Kind, "uint8", 8}
	Uint16 DataType = &fixed{Uint16Kind, "uint16", 16}
	Uint32 DataType = &fixed{Uint32Kind, "uint32", 32}
	Uint64 DataType = &fixed{Uint64Kind, "uint64", 64}

	Float32 DataType = &fixed{Float32Kind, "float32", 32}
	Float64 DataType = &fixed{Float64Kind, "float64", 64}

	// String and Binary use the Arrow variable size binary view layout, which
	// is the reasoning in document 02: a sixteen byte view per value holding
	// the length, a four byte inline prefix, and then either the rest of the
	// value inline for twelve bytes or fewer or a buffer index and offset for
	// longer ones.
	String DataType = &variable{StringKind, "string"}
	Binary DataType = &variable{BinaryKind, "binary"}

	// LargeString and LargeBinary are the classic offsets and data layout with
	// 64 bit offsets. They are kept for interoperability, since that is what
	// arrives over Arrow IPC, and converted at the boundary.
	LargeString DataType = &variable{LargeStringKind, "large_string"}
	LargeBinary DataType = &variable{LargeBinaryKind, "large_binary"}

	// Date32 is days since the Unix epoch and Date64 is milliseconds since it,
	// constrained to exact multiples of a day. Date32 is the one to use.
	Date32 DataType = &fixed{Date32Kind, "date32", 32}
	Date64 DataType = &fixed{Date64Kind, "date64", 64}
)

The parameterless types. These are pointers to values with unexported fields, so the types themselves cannot be modified. Do not reassign the variables.

func Coerce 

func Coerce(a, b DataType) (DataType, error)

Coerce returns the type that two columns have in common, or an error saying they have none.

It is strict on purpose. Two int64 columns give an int64. An int64 column and a float64 column give an error, not a float64, even though every int64 has a float64 near it. This is the rule from document 02 and it is the single biggest correctness difference from pandas, where the upcast happens quietly and an id column that fit exactly in an int64 comes out the other side as a float64 with the low bits rounded off. Polars is strict here and is right to be. The error arrives when the plan is built, before any data is read, and it names the cast the caller has to write.

The exceptions are the cases where nothing can be lost:

A null column combines with anything and takes the other type. Every value in it is missing, so there is nothing to convert.

A dictionary combines with its own value type and with another dictionary over that value type. Dictionary encoding is how the values are stored, not what they are, so a dictionary of strings and a column of strings are the same column written two ways. Two dictionaries keep the encoding and take whichever index type holds both.

The nested types combine element by element, so a list of null and a list of int64 give a list of int64, which is the case that turns up whenever an empty list is read from JSON.

Coerce is for two columns. A literal follows different rules, because the caller wrote it and it has no storage to preserve. See CoerceLiteral.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	// An int64 column and a float64 column have nothing in common that keeps
	// every value, so the caller is told to say which one they meant. pandas
	// would return float64 here and round the low bits off any id large enough
	// to matter.
	fmt.Println(dtype.Coerce(dtype.Int64, dtype.Float64))

	// A column read as all nulls, which is what an empty JSON array gives,
	// takes the type of whatever it is concatenated with.
	fmt.Println(dtype.Coerce(dtype.List{Elem: dtype.Null}, dtype.List{Elem: dtype.Int64}))

	// Dictionary encoding is how the values are stored and not what they are.
	fmt.Println(dtype.Coerce(dtype.Dictionary{Index: dtype.Uint32, Value: dtype.String}, dtype.String))
}

Output:
<nil> dtype: cannot combine int64 and float64, cast one side explicitly
list<int64> <nil>
string <nil>

func CoerceLiteral 

func CoerceLiteral(column, literal DataType) (DataType, error)

CoerceLiteral returns the type a comparison or an arithmetic operation between a column and a literal is carried out in.

A literal is looser than a column because there is nothing to preserve. The caller wrote 1 in their own source and meant the number, not an int64, so the literal takes the column's type wherever that is exact and the column keeps its own storage. Writing df.Col("count").Gt(0) should not be a type error and should not quietly turn a uint32 column into an int64 one.

It works on types, so it says whether a literal of that type can take the column's type at all. Whether the particular value fits, meaning whether 300 fits in the int8 column it is being compared against, is a question about the value and is answered by the layer that has it.

A float literal against an integer column is an error rather than a truncation, because 1.5 has no int64 spelling and rounding it silently is the same class of mistake as upcasting the column silently. The same goes for a float literal against a decimal column, where being exact is the entire reason the column is a decimal.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	// Comparing a uint32 column against the literal 0 is not a type error and
	// does not widen the column.
	fmt.Println(dtype.CoerceLiteral(dtype.Uint32, dtype.Int64))

	// A float literal against an integer column is refused, because 1.5 has no
	// int64 spelling and rounding it quietly is the mistake this package is
	// trying not to make.
	fmt.Println(dtype.CoerceLiteral(dtype.Int64, dtype.Float64))
}

Output:
uint32 <nil>
<nil> dtype: cannot use a float64 literal with a int64 column, cast the column or write a int64 literal

type Decimal128 

type Decimal128 struct {
	Precision int32
	Scale     int32
}

Decimal128 is an exact fixed point number stored as a 128 bit signed integer scaled by ten to the power of Scale.

Precision is the total number of significant digits and Scale is how many of them are after the decimal point, so money in pounds and pence is Decimal128{Precision: 18, Scale: 2} and the value 12.34 is stored as 1234.

This is what a currency column should be. A float64 cannot represent 0.1, so summing a million float prices gives an answer that is close to right and disagrees with the ledger, and the report gets rewritten in the accounting system instead.

Scale may be negative, which multiplies rather than divides: a Scale of -3 means the stored integer counts thousands. Arrow allows this and readers in the wild produce it.

func (Decimal128) Bits 

func (t Decimal128) Bits() int

Bits returns 128.

func (Decimal128) Kind 

func (t Decimal128) Kind() Kind

Kind returns Decimal128Kind.

func (Decimal128) String 

func (t Decimal128) String() string

String returns the canonical name, such as "decimal128(18, 2)".

type Decimal256 

type Decimal256 struct {
	Precision int32
	Scale     int32
}

Decimal256 is an exact fixed point number stored as a 256 bit signed integer scaled by ten to the power of Scale. It is Decimal128 with room for more digits, for the cases where 38 is not enough.

func (Decimal256) Bits 

func (t Decimal256) Bits() int

Bits returns 256.

func (Decimal256) Kind 

func (t Decimal256) Kind() Kind

Kind returns Decimal256Kind.

func (Decimal256) String 

func (t Decimal256) String() string

String returns the canonical name, such as "decimal256(50, 8)".

type Dictionary 

type Dictionary struct {
	Index DataType
	Value DataType
}

Dictionary stores each distinct value once and one Index per row pointing at it, which is what a low cardinality string column wants: a million rows of twenty country names become a million int32 values and twenty strings.

This is the equivalent of a pandas categorical, except that the encoding is storage rather than semantics. A dictionary of strings compares, sorts and groups exactly like a column of strings. Ordered categories, where the comparison follows the dictionary order rather than the value order, are a separate thing and are not this.

Index must be an integer type. Uint32 is the usual choice, and a smaller one is worth it when the cardinality is known to be small.

func (Dictionary) Kind 

func (t Dictionary) Kind() Kind

Kind returns DictionaryKind.

func (Dictionary) String 

func (t Dictionary) String() string

String returns the canonical name, such as "dictionary<uint32, string>".

type Duration 

type Duration struct {
	Unit TimeUnit
}

Duration is an elapsed span of time, stored as an int64 count of Unit.

It is the difference between two timestamps and it is exact, unlike Interval, which is calendar arithmetic.

func (Duration) Bits 

func (t Duration) Bits() int

Bits returns 64.

func (Duration) Kind 

func (t Duration) Kind() Kind

Kind returns DurationKind.

func (Duration) String 

func (t Duration) String() string

String returns the canonical name, such as "duration[ns]".

type Field 

type Field struct {
	Name     string
	Type     DataType
	Nullable bool
	Metadata Metadata
}

Field is one named column in a schema, or one named member of a struct type.

Nullable is on the field rather than on the type. That is the split that keeps int64 meaning int64 whether or not a value happens to be missing, which is the thing pandas got wrong and spent a decade adding nullable dtypes to work around. A non-nullable field is a promise about the data, and the builders and readers check it.

func (Field) Equal 

func (f Field) Equal(other Field) bool

Equal reports whether f and other have the same name, type, nullability and metadata.

func (Field) String 

func (f Field) String() string

String returns the field as it appears inside a printed schema, such as "price: float64 not null".

func (Field) Validate 

func (f Field) Validate() error

Validate reports whether f has a name and a valid type.

type FixedSizeBinary 

type FixedSizeBinary struct {
	ByteWidth int32
}

FixedSizeBinary is ByteWidth bytes per value with no offsets, which is what a UUID, a hash or a fixed length identifier should be. It is the only binary type that is fixed width.

func (FixedSizeBinary) Bits 

func (t FixedSizeBinary) Bits() int

Bits returns the width of one value in bits, which is eight times ByteWidth.

func (FixedSizeBinary) Kind 

func (t FixedSizeBinary) Kind() Kind

Kind returns FixedSizeBinaryKind.

func (FixedSizeBinary) String 

func (t FixedSizeBinary) String() string

String returns the canonical name, such as "fixed_size_binary[16]".

type FixedSizeList 

type FixedSizeList struct {
	Elem DataType
	Len  int32
}

FixedSizeList is exactly Len elements of Elem per row, with no offsets. It is how a column of coordinates or a column of small embeddings should be stored, since the offsets in a List would all be multiples of the same number.

func (FixedSizeList) Kind 

func (t FixedSizeList) Kind() Kind

Kind returns FixedSizeListKind.

func (FixedSizeList) String 

func (t FixedSizeList) String() string

String returns the canonical name, such as "fixed_size_list<float32>[3]".

type FixedWidth 

type FixedWidth interface {
	DataType

	// Bits returns the width of one value in bits.
	Bits() int
}

FixedWidth is implemented by the types whose values all take the same number of bits, which is the property that lets a kernel index into a buffer by multiplication rather than by following offsets.

Bool is fixed width at one bit, not one byte. Anything sizing a buffer has to handle that, and it is the reason this reports bits rather than bytes.

The variable width string and binary types are not here even though their view structs are a fixed sixteen bytes, because the bytes those views point at are not.

type Interval 

type Interval struct {
	Unit IntervalUnit
}

Interval is a span expressed in calendar units, which do not have a fixed length. See IntervalUnit for what each one counts and how wide it is.

func (Interval) Bits 

func (t Interval) Bits() int

Bits returns the storage width, which depends on the unit: 32 for YearMonth, 64 for DayTime and 128 for MonthDayNano.

func (Interval) Kind 

func (t Interval) Kind() Kind

Kind returns IntervalKind.

func (Interval) String 

func (t Interval) String() string

String returns the canonical name, such as "interval[month_day_nano]".

type IntervalUnit 

type IntervalUnit uint8

IntervalUnit is what an interval counts.

An interval is calendar arithmetic rather than a fixed span of time, which is why it is a separate type from Duration. One month is not a number of seconds, and adding one month to the 31st of January has to land somewhere the caller agrees with. 

const (
	// YearMonth counts whole months, stored as one int32.
	YearMonth IntervalUnit = iota

	// DayTime counts days and milliseconds, stored as two int32 values.
	DayTime

	// MonthDayNano counts months, days and nanoseconds, stored as an int32, an
	// int32 and an int64. It is the one that can express everything the other
	// two can.
	MonthDayNano
)

The interval units.

func (IntervalUnit) String 

func (u IntervalUnit) String() string

String returns the unit's name.

func (IntervalUnit) Valid 

func (u IntervalUnit) Valid() bool

Valid reports whether u is one of the three defined units.

type KeyValue 

type KeyValue struct {
	Key   string
	Value string
}

KeyValue is one piece of metadata.

type Kind 

type Kind uint8

Kind is a type with its parameters removed.

The zero Kind is InvalidKind rather than a real type, so that a Kind that was never set does not silently read as null or as bool. 

const (
	InvalidKind Kind = iota
	NullKind
	BoolKind
	Int8Kind
	Int16Kind
	Int32Kind
	Int64Kind
	Uint8Kind
	Uint16Kind
	Uint32Kind
	Uint64Kind
	Float32Kind
	Float64Kind
	StringKind
	BinaryKind
	LargeStringKind
	LargeBinaryKind
	FixedSizeBinaryKind
	Date32Kind
	Date64Kind
	Time32Kind
	Time64Kind
	TimestampKind
	DurationKind
	IntervalKind
	Decimal128Kind
	Decimal256Kind
	ListKind
	LargeListKind
	FixedSizeListKind
	StructKind
	MapKind
	DictionaryKind
)

The kinds. Every DataType reports exactly one of these.

func (Kind) String 

func (k Kind) String() string

String returns the kind's name, which is the type's name with the parameters left off. The kind of a microsecond timestamp prints as "timestamp".

type LargeList 

type LargeList struct {
	Elem DataType
}

LargeList is List with 64 bit offsets, for a child array with more than two billion elements in total.

func (LargeList) Kind 

func (t LargeList) Kind() Kind

Kind returns LargeListKind.

func (LargeList) String 

func (t LargeList) String() string

String returns the canonical name, such as "large_list<int64>".

type List 

type List struct {
	Elem DataType
}

List is a variable length sequence of Elem per row, stored as 32 bit offsets into one child array. Every element of every row lives in the same child, so a kernel that does not care about the row boundaries can run over the child directly.

func (List) Kind 

func (t List) Kind() Kind

Kind returns ListKind.

func (List) String 

func (t List) String() string

String returns the canonical name, such as "list<int64>".

type Map 

type Map struct {
	Key   DataType
	Value DataType
}

Map is a variable number of key and value pairs per row.

It is stored as a list of two field structs, which is worth knowing because it means the keys of every row are one contiguous array. Keys are not deduplicated across rows and nothing here enforces that they are unique within a row, since checking that on every read would cost more than the type is worth.

func (Map) Kind 

func (t Map) Kind() Kind

Kind returns MapKind.

func (Map) String 

func (t Map) String() string

String returns the canonical name, such as "map<string, int64>".

type Metadata 

type Metadata []KeyValue

Metadata is a list of key and value pairs carried alongside a field or a schema.

It is a slice rather than a map because it round trips through Arrow IPC and Parquet, both of which specify an ordered list, and because a map would reorder the pairs on every write and make two identical schemas produce two different files. Duplicate keys are not rejected, since the formats allow them, and Get returns the first.

Nothing in kuma reads metadata. It is here so that a value written by another tool survives a round trip through a kuma program, and so that a caller can attach a unit or a description that their own code understands.

func (Metadata) Clone 

func (m Metadata) Clone() Metadata

Clone returns a copy that shares no storage with m.

func (Metadata) Equal 

func (m Metadata) Equal(other Metadata) bool

Equal reports whether m and other hold the same pairs in the same order.

func (Metadata) Get 

func (m Metadata) Get(key string) (string, bool)

Get returns the value for the first pair with the given key, and whether there was one.

type Schema 

type Schema struct {
	Fields   []Field
	Metadata Metadata
}

Schema is the ordered list of fields in a frame.

The order is part of the schema. Two schemas with the same fields in a different order are different schemas, because column order is what a positional read of a CSV or a Parquet row group depends on, and because printing a frame has to put the columns somewhere.

Field names are not required to be unique by the type itself, since a CSV with two columns called "id" is a real thing that has to be readable. Validate is what rejects duplicates, and the frame layer calls it.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	orders := dtype.Schema{
		Fields: []dtype.Field{
			{Name: "id", Type: dtype.Int64},
			{Name: "customer", Type: dtype.Dictionary{Index: dtype.Uint32, Value: dtype.String}},
			{Name: "placed", Type: dtype.Timestamp{Unit: dtype.Microsecond, Zone: "UTC"}},
			{Name: "total", Type: dtype.Decimal128{Precision: 18, Scale: 2}, Nullable: true},
		},
	}

	if err := orders.Validate(); err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(orders)
}

Output:
schema<id: int64 not null, customer: dictionary<uint32, string> not null, placed: timestamp[us, tz=UTC] not null, total: decimal128(18, 2)>

func (Schema) Clone 

func (s Schema) Clone() Schema

Clone returns a copy that shares no slice storage with s. The types themselves are not copied, since they are immutable.

func (Schema) Equal 

func (s Schema) Equal(other Schema) bool

Equal reports whether s and other have the same fields in the same order with the same metadata.

func (Schema) Field 

func (s Schema) Field(name string) (Field, bool)

Field returns the first field with the given name and whether there was one.

func (Schema) Index 

func (s Schema) Index(name string) int

Index returns the position of the first field with the given name, or -1 if there is none.

func (Schema) Len 

func (s Schema) Len() int

Len returns the number of fields.

func (Schema) Names 

func (s Schema) Names() []string

Names returns the field names in order.

func (Schema) Select 

func (s Schema) Select(names ...string) (Schema, error)

Select returns a schema holding the named fields in the order given. It reports an error naming the first field that is not in s, along with the names that are, since the usual cause is a typo or a stale column name.

Metadata on the schema is carried over. Selecting a subset of the columns does not change what the table as a whole is.

Example 
package main

import (
	"fmt"

	"github.com/tamnd/kuma/dtype"
)

func main() {
	orders := dtype.Schema{Fields: []dtype.Field{
		{Name: "id", Type: dtype.Int64},
		{Name: "customer", Type: dtype.String},
		{Name: "total", Type: dtype.Float64},
	}}

	narrow, err := orders.Select("customer", "total")
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(narrow.Names())

	// A name that is not there says so, and says what is.
	_, err = orders.Select("totl")
	fmt.Println(err)
}

Output:
[customer total]
dtype: no field named "totl", have id, customer, total

func (Schema) String 

func (s Schema) String() string

String returns the schema on one line, such as "schema<id: int64 not null, name: string>".

func (Schema) Validate 

func (s Schema) Validate() error

Validate reports whether s is a schema a frame can be built from, meaning every field has a name, no two fields share one, and every type is valid.

Duplicate names are rejected here rather than by the Schema type itself, because a CSV with two columns called "id" is a real file, and a reader has to be able to describe it before it can rename anything.

type Struct 

type Struct struct {
	Fields []Field
}

Struct is a fixed set of named fields per row, each stored as its own child array. A struct column of three fields is three columns that share one validity bitmap, so selecting one field costs nothing.

func (Struct) Field 

func (t Struct) Field(name string) (Field, bool)

Field returns the field with the given name and whether it was found.

func (Struct) Kind 

func (t Struct) Kind() Kind

Kind returns StructKind.

func (Struct) String 

func (t Struct) String() string

String returns the canonical name, such as "struct<a: int64 not null, b: string>".

Nullability is printed because it is part of what makes two struct types different, and leaving it out would mean two types that are not equal print the same way.

type Time32 

type Time32 struct {
	Unit TimeUnit
}

Time32 is a time of day with no date, stored in 32 bits as a count of Unit since midnight. Only Second and Millisecond fit, which Validate checks.

func (Time32) Bits 

func (t Time32) Bits() int

Bits returns 32.

func (Time32) Kind 

func (t Time32) Kind() Kind

Kind returns Time32Kind.

func (Time32) String 

func (t Time32) String() string

String returns the canonical name, such as "time32[s]".

type Time64 

type Time64 struct {
	Unit TimeUnit
}

Time64 is a time of day with no date, stored in 64 bits as a count of Unit since midnight. Only Microsecond and Nanosecond are allowed, so that there is exactly one representation of every resolution across the two time types.

func (Time64) Bits 

func (t Time64) Bits() int

Bits returns 64.

func (Time64) Kind 

func (t Time64) Kind() Kind

Kind returns Time64Kind.

func (Time64) String 

func (t Time64) String() string

String returns the canonical name, such as "time64[ns]".

type TimeUnit 

type TimeUnit uint8

TimeUnit is the resolution of a temporal value.

The zero value is Second, which is the coarsest unit and the one that cannot silently lose data by being wrong: reading a value that is really in nanoseconds as if it were seconds gives an obviously absurd date rather than a plausible wrong one. 

const (
	Second TimeUnit = iota
	Millisecond
	Microsecond
	Nanosecond
)

The time units.

func (TimeUnit) String 

func (u TimeUnit) String() string

String returns the unit's short name, which is the one Arrow uses in a type name: s, ms, us or ns.

func (TimeUnit) Valid 

func (u TimeUnit) Valid() bool

Valid reports whether u is one of the four defined units.

type Timestamp 

type Timestamp struct {
	Unit TimeUnit
	Zone string
}

Timestamp is an instant, stored as an int64 count of Unit since the Unix epoch.

Zone is an IANA name such as "Europe/London", or empty. The distinction is not a display detail. An empty Zone means the value is naive local time with no instant attached, so two of them cannot be compared across zones and truncating to a day is unambiguous. A non-empty Zone means the int64 is a real instant in UTC and the zone says how to render it and how calendar arithmetic on it behaves around a daylight saving transition.

Nothing in this package resolves Zone against the tzdata database, because a binary built without tzdata would then reject a schema that is perfectly valid on the machine that wrote it. Resolution happens where the arithmetic happens.

func (Timestamp) Bits 

func (t Timestamp) Bits() int

Bits returns 64.

func (Timestamp) Kind 

func (t Timestamp) Kind() Kind

Kind returns TimestampKind.

func (Timestamp) String 

func (t Timestamp) String() string

String returns the canonical name, such as "timestamp[us]" or "timestamp[us, tz=UTC]".

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.