zed

package module
Version: v1.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 27, 2023 License: BSD-3-Clause Imports: 20 Imported by: 21

README

Zed Tests GoPkg

Zed offers a new approach to data that makes it easier to manipulate and manage your data.

With Zed's new super-structured data model, messy JSON data can easily be given the fully-typed precision of relational tables without giving up JSON's uncanny ability to represent eclectic data.

Trying out Zed is easy: just install the command-line tool zq.

zq is a lot like jq but is built from the ground up as a search and analytics engine based on the Zed data model. Since Zed data is a proper superset of JSON, zq also works natively with JSON.

While zq and the Zed data formats are production quality, the Zed project's Zed data lake is a bit earlier in development.

For a non-technical user, Zed is as easy to use as web search while for a technical user, Zed exposes its technical underpinnings in a gradual slope, providing as much detail as desired, packaged up in the easy-to-understand ZSON data format and Zed language.

Why?

We think data is hard and it should be much, much easier.

While schemas are a great way to model and organize your data, they often get in the way when you are just trying to store or transmit your semi-structured data.

Also, why should you have to set up one system for search and another completely different system for historical analytics? And the same unified search/analytics system that works at cloud scale should run easily as a lightweight command-line tool on your laptop.

And rather than having to set up complex ETL pipelines with brittle transformation logic, managing your data lake should be as easy as git.

Finally, we believe a lightweight data store that provides easy search and analytics would be a great place to store data sets for data science and data engineering experiments running in Python and providing easy integration with your favorite Python libraries.

How?

Zed solves all these problems with a new foundational data format called ZSON, which is a superset of JSON and the relational models. ZSON is syntax-compatible with JSON but it has a comprehensive type system that you can use as little or as much as you like. Zed types can be used as schemas.

The Zed language offers a gentle learning curve, which spans the gamut from simple keyword search to powerful data-transformation operators like lateral sub-queries and shaping.

Zed also has a cloud-based object design that was modeled after the git design pattern. Commits to the lake are transactional and consistent. Search index updates are also transactionally consistent with any ingested data, and searches can run with or without indexes.

Quick Start

Check out the installation page for a quick and easy install.

Detailed documentation for the entire Zed system and language is available on the Zed docs site.

Zui

The Zui app is an Electron-based desktop app to explore, query, and shape data in your Zed lake.

We originally developed Zui for security-oriented use cases (having tight integration with Zeek, Suricata, and Wireshark), but we are actively extending Zui with UX for handling generic data sets to support data science, data engineering, and ETL use cases.

Contributing

See the contributing guide on how you can help improve Zed!

Join the Community

Join our public Slack workspace for announcements, Q&A, and to trade tips!

Acknowledgment

We modeled this README after Philip O'Toole's brilliantly succinct description of rqlite.

Documentation

Overview

Package zng implements a data typing system based on the zeek type system. All zeek types are defined here and implement the Type interface while instances of values implement the Value interface. All values conform to exactly one type. The package provides a fast-path for comparing a value to a byte slice without having to create a zeek value from the byte slice. To exploit this, all values include a Comparison method that returns a Predicate function that takes a byte slice and a Type and returns a boolean indicating whether the the byte slice with the indicated Type matches the value. The package also provides mechanism for coercing values in well-defined and natural ways.

Index

Constants

View Source
const (
	MaxEnumSymbols  = 100_000
	MaxRecordFields = 100_000
	MaxUnionTypes   = 100_000
)
View Source
const (
	IDUint8       = 0
	IDUint16      = 1
	IDUint32      = 2
	IDUint64      = 3
	IDUint128     = 4
	IDUint256     = 5
	IDInt8        = 6
	IDInt16       = 7
	IDInt32       = 8
	IDInt64       = 9
	IDInt128      = 10
	IDInt256      = 11
	IDDuration    = 12
	IDTime        = 13
	IDFloat16     = 14
	IDFloat32     = 15
	IDFloat64     = 16
	IDFloat128    = 17
	IDFloat256    = 18
	IDDecimal32   = 19
	IDDecimal64   = 20
	IDDecimal128  = 21
	IDDecimal256  = 22
	IDBool        = 23
	IDBytes       = 24
	IDString      = 25
	IDIP          = 26
	IDNet         = 27
	IDType        = 28
	IDNull        = 29
	IDTypeComplex = 30
)
View Source
const (
	TypeValueRecord  = 30
	TypeValueArray   = 31
	TypeValueSet     = 32
	TypeValueMap     = 33
	TypeValueUnion   = 34
	TypeValueEnum    = 35
	TypeValueError   = 36
	TypeValueNameDef = 37
	TypeValueNameRef = 38
	TypeValueMax     = TypeValueNameRef
)

Variables

View Source
var (
	ErrNotArray  = errors.New("cannot index a non-array")
	ErrIndex     = errors.New("array index out of bounds")
	ErrUnionTag  = errors.New("invalid union tag")
	ErrEnumIndex = errors.New("enum index out of bounds")
)
View Source
var (
	TypeUint8    = &TypeOfUint8{}
	TypeUint16   = &TypeOfUint16{}
	TypeUint32   = &TypeOfUint32{}
	TypeUint64   = &TypeOfUint64{}
	TypeInt8     = &TypeOfInt8{}
	TypeInt16    = &TypeOfInt16{}
	TypeInt32    = &TypeOfInt32{}
	TypeInt64    = &TypeOfInt64{}
	TypeDuration = &TypeOfDuration{}
	TypeTime     = &TypeOfTime{}
	TypeFloat16  = &TypeOfFloat16{}
	TypeFloat32  = &TypeOfFloat32{}
	TypeFloat64  = &TypeOfFloat64{}
	// XXX add TypeDecimal
	TypeBool   = &TypeOfBool{}
	TypeBytes  = &TypeOfBytes{}
	TypeString = &TypeOfString{}
	TypeIP     = &TypeOfIP{}
	TypeNet    = &TypeOfNet{}
	TypeType   = &TypeOfType{}
	TypeNull   = &TypeOfNull{}
)
View Source
var (
	ErrMissingField = errors.New("record missing a field")
	ErrNotContainer = errors.New("expected container type, got primitive")
)
View Source
var (
	NullUint8    = &Value{Type: TypeUint8}
	NullUint16   = &Value{Type: TypeUint16}
	NullUint32   = &Value{Type: TypeUint32}
	NullUint64   = &Value{Type: TypeUint64}
	NullInt8     = &Value{Type: TypeInt8}
	NullInt16    = &Value{Type: TypeInt16}
	NullInt32    = &Value{Type: TypeInt32}
	NullInt64    = &Value{Type: TypeInt64}
	NullDuration = &Value{Type: TypeDuration}
	NullTime     = &Value{Type: TypeTime}
	NullFloat16  = &Value{Type: TypeFloat16}
	NullFloat32  = &Value{Type: TypeFloat32}
	NullFloat64  = &Value{Type: TypeFloat64}
	NullBool     = &Value{Type: TypeBool}
	NullBytes    = &Value{Type: TypeBytes}
	NullString   = &Value{Type: TypeString}
	NullIP       = &Value{Type: TypeIP}
	NullNet      = &Value{Type: TypeNet}
	NullType     = &Value{Type: TypeType}
	Null         = &Value{Type: TypeNull}

	False = NewBool(false)
	True  = NewBool(true)
)
View Source
var ErrMissing = errors.New("missing")

ErrMissing is a Go error that implies a missing value in the runtime logic whereas Missing is a Zed error value that represents a missing value embedded in the dataflow computation.

View Source
var Missing = zcode.Bytes("missing")

Missing is value that represents an error condition arising from a referenced entity not present, e.g., a reference to a non-existent record field, a map lookup for a key not present, an array index that is out of range, etc. The Missing error can be propagated through functions and expressions and each operator has clearly defined semantics with respect to the Missing value. For example, "true AND MISSING" is MISSING.

View Source
var Quiet = zcode.Bytes("quiet")
View Source
var SkipContainer = errors.New("skip this container")

SkipContainer is used as a return value from Visitors to indicate that the container passed in the call should not be visited. It is not returned as an error by any function.

Functions

func AppendBool

func AppendBool(zb zcode.Bytes, b bool) zcode.Bytes

func AppendDuration

func AppendDuration(bytes zcode.Bytes, d nano.Duration) zcode.Bytes

func AppendFloat16 added in v1.5.0

func AppendFloat16(zb zcode.Bytes, f float32) zcode.Bytes

func AppendFloat32

func AppendFloat32(zb zcode.Bytes, f float32) zcode.Bytes

func AppendFloat64

func AppendFloat64(zb zcode.Bytes, d float64) zcode.Bytes

func AppendIP

func AppendIP(zb zcode.Bytes, a netip.Addr) zcode.Bytes

func AppendInt

func AppendInt(bytes zcode.Bytes, i int64) zcode.Bytes

func AppendNet

func AppendNet(zb zcode.Bytes, p netip.Prefix) zcode.Bytes

func AppendTime

func AppendTime(bytes zcode.Bytes, t nano.Ts) zcode.Bytes

func AppendTypeValue added in v1.0.0

func AppendTypeValue(b zcode.Bytes, t Type) zcode.Bytes

func AppendUint

func AppendUint(bytes zcode.Bytes, i uint64) zcode.Bytes

func BuildUnion

func BuildUnion(b *zcode.Builder, tag int, val zcode.Bytes)

BuildUnion appends to b a union described by tag and val.

func CompareTypes added in v1.0.0

func CompareTypes(a, b Type) int

func DecodeBool

func DecodeBool(zv zcode.Bytes) bool

func DecodeBytes

func DecodeBytes(zv zcode.Bytes) []byte

func DecodeDuration

func DecodeDuration(zv zcode.Bytes) nano.Duration

func DecodeError

func DecodeError(zv zcode.Bytes) error

func DecodeFloat

func DecodeFloat(zb zcode.Bytes) float64

func DecodeFloat16 added in v1.5.0

func DecodeFloat16(zb zcode.Bytes) float32

func DecodeFloat32

func DecodeFloat32(zb zcode.Bytes) float32

func DecodeFloat64

func DecodeFloat64(zv zcode.Bytes) float64

func DecodeIP

func DecodeIP(zv zcode.Bytes) netip.Addr

func DecodeInt

func DecodeInt(zv zcode.Bytes) int64

func DecodeLength added in v1.0.0

func DecodeLength(tv zcode.Bytes) (int, zcode.Bytes)

func DecodeName added in v1.0.0

func DecodeName(tv zcode.Bytes) (string, zcode.Bytes)

func DecodeNet

func DecodeNet(zv zcode.Bytes) netip.Prefix

func DecodeString

func DecodeString(zv zcode.Bytes) string

func DecodeTime

func DecodeTime(zv zcode.Bytes) nano.Ts

func DecodeUint

func DecodeUint(zv zcode.Bytes) uint64

func EncodeBool

func EncodeBool(b bool) zcode.Bytes

func EncodeBytes

func EncodeBytes(b []byte) zcode.Bytes

func EncodeDuration

func EncodeDuration(d nano.Duration) zcode.Bytes

func EncodeError

func EncodeError(err error) zcode.Bytes

func EncodeFloat16 added in v1.5.0

func EncodeFloat16(d float32) zcode.Bytes

func EncodeFloat32

func EncodeFloat32(d float32) zcode.Bytes

func EncodeFloat64

func EncodeFloat64(d float64) zcode.Bytes

func EncodeIP

func EncodeIP(a netip.Addr) zcode.Bytes

func EncodeInt

func EncodeInt(i int64) zcode.Bytes

func EncodeNet

func EncodeNet(p netip.Prefix) zcode.Bytes

func EncodeString

func EncodeString(s string) zcode.Bytes

func EncodeTime

func EncodeTime(t nano.Ts) zcode.Bytes

func EncodeTypeValue

func EncodeTypeValue(t Type) zcode.Bytes

func EncodeUint

func EncodeUint(i uint64) zcode.Bytes

func IsContainerType

func IsContainerType(typ Type) bool

func IsFloat

func IsFloat(id int) bool

True iff the type id is encoded as a float encoding. XXX add IDDecimal here when we implement coercible math with it.

func IsInteger

func IsInteger(id int) bool

True iff the type id is encoded as a zng signed or unsigened integer zcode.Bytes.

func IsNumber

func IsNumber(id int) bool

True iff the type id is encoded as a zng signed or unsigned integer zcode.Bytes, float16 zcode.Bytes, float32 zcode.Bytes, or float64 zcode.Bytes.

func IsPrimitiveType

func IsPrimitiveType(typ Type) bool

func IsRecordType

func IsRecordType(typ Type) bool

func IsSigned

func IsSigned(id int) bool

True iff the type id is encoded as a number encoding and is signed.

func IsUnionType

func IsUnionType(typ Type) bool

func IsUnsigned added in v1.9.0

func IsUnsigned(id int) bool

True iff the type id is encoded as a number encoding and is unsigned.

func LeadingOnes added in v1.4.0

func LeadingOnes(b []byte) int

LeadingOnes returns the number of leading one bits in b.

func NormalizeMap

func NormalizeMap(zv zcode.Bytes) zcode.Bytes

NormalizeMap interprets zv as a map body and returns an equivalent map body that is normalized according to the ZNG specification (i.e., the tag-counted value of each entry's key is lexicographically greater than that of the preceding entry).

func NormalizeSet

func NormalizeSet(zv zcode.Bytes) zcode.Bytes

NormalizeSet interprets zv as a set body and returns an equivalent set body that is normalized according to the ZNG specification (i.e., each element's tag-counted value is lexicographically greater than that of the preceding element).

func PrimitiveName added in v1.0.0

func PrimitiveName(typ Type) string

func TypeID

func TypeID(typ Type) int

func Walk

func Walk(typ Type, body zcode.Bytes, visit Visitor) error

Types

type Allocator added in v1.0.0

type Allocator interface {
	NewValue(Type, zcode.Bytes) *Value
	CopyValue(Value) *Value
}

type Context

type Context struct {
	// contains filtered or unexported fields
}

A Context implements the "type context" in the Zed model. For a given set of related Values, each Value has a type from a shared Context. The Context manages the transitive closure of Types so that each unique type corresponds to exactly one Type pointer allowing type equivlance to be determined by pointer comparison. (Type pointers from distinct Contexts obviously do not have this property.) A Context also provides an efficient means to translate type values (represented as serialized ZNG) to Types. This provides an efficient means to translate Type pointers from one context to another.

func NewContext

func NewContext() *Context

func (*Context) AddFields added in v1.5.0

func (c *Context) AddFields(r *Value, newFields []Field, vals []Value) (*Value, error)

AddFields returns a new Record with fields equal to the given record along with new rightmost fields as indicated with the given values. If any of the newly provided fieldss already exists in the specified value, an error is returned.

func (*Context) DecodeTypeValue added in v1.0.0

func (c *Context) DecodeTypeValue(tv zcode.Bytes) (Type, zcode.Bytes)

func (*Context) LookupByValue

func (c *Context) LookupByValue(tv zcode.Bytes) (Type, error)

LookupByValue returns the Type indicated by a binary-serialized type value. This provides a means to translate a type-context-independent serialized encoding for an arbitrary type into the reciever Context.

func (*Context) LookupType

func (c *Context) LookupType(id int) (Type, error)

func (*Context) LookupTypeArray

func (c *Context) LookupTypeArray(inner Type) *TypeArray

func (*Context) LookupTypeDef

func (c *Context) LookupTypeDef(name string) *TypeNamed

LookupTypeDef returns the named type last bound to name by LookupTypeNamed. It returns nil if name is unbound.

func (*Context) LookupTypeEnum

func (c *Context) LookupTypeEnum(symbols []string) *TypeEnum

func (*Context) LookupTypeError added in v1.0.0

func (c *Context) LookupTypeError(inner Type) *TypeError

func (*Context) LookupTypeMap

func (c *Context) LookupTypeMap(keyType, valType Type) *TypeMap

func (*Context) LookupTypeNamed added in v1.0.0

func (c *Context) LookupTypeNamed(name string, inner Type) (*TypeNamed, error)

LookupTypeNamed returns the named type for name and inner. It also binds name to that named type. LookupTypeNamed returns an error if name is not a valid UTF-8 string or is a primitive type name.

func (*Context) LookupTypeRecord

func (c *Context) LookupTypeRecord(fields []Field) (*TypeRecord, error)

LookupTypeRecord returns a TypeRecord within this context that binds with the indicated fields. Subsequent calls with the same fields will return the same record pointer. If the type doesn't exist, it's created, stored, and returned. The closure of types within the fields must all be from this type context. If you want to use fields from a different type context, use TranslateTypeRecord.

func (*Context) LookupTypeSet

func (c *Context) LookupTypeSet(inner Type) *TypeSet

func (*Context) LookupTypeUnion

func (c *Context) LookupTypeUnion(types []Type) *TypeUnion

func (*Context) LookupTypeValue

func (c *Context) LookupTypeValue(typ Type) *Value

func (*Context) Missing added in v1.0.0

func (c *Context) Missing() *Value

func (*Context) MustLookupTypeRecord

func (c *Context) MustLookupTypeRecord(fields []Field) *TypeRecord

func (*Context) NewError added in v1.0.0

func (c *Context) NewError(err error) *Value

func (*Context) NewErrorf added in v1.0.0

func (c *Context) NewErrorf(format string, args ...interface{}) *Value

func (*Context) Quiet added in v1.0.0

func (c *Context) Quiet() *Value

func (*Context) Reset

func (c *Context) Reset()

func (*Context) StringTypeError added in v1.0.0

func (c *Context) StringTypeError() *TypeError

func (*Context) TranslateType

func (c *Context) TranslateType(ext Type) (Type, error)

TranslateType takes a type from another context and creates and returns that type in this context.

func (*Context) WrapError added in v1.0.0

func (c *Context) WrapError(msg string, val *Value) *Value

type DuplicateFieldError added in v1.0.0

type DuplicateFieldError struct {
	Name string
}

func (*DuplicateFieldError) Error added in v1.0.0

func (d *DuplicateFieldError) Error() string

type Field added in v1.5.0

type Field struct {
	Name string
	Type Type
}

Field defines the name and type of a field for TypeRecord.

func NewField added in v1.5.0

func NewField(name string, typ Type) Field

type Kind added in v1.0.0

type Kind int
const (
	PrimitiveKind Kind = iota
	RecordKind
	ArrayKind
	SetKind
	MapKind
	UnionKind
	EnumKind
	ErrorKind
)

func (Kind) String added in v1.0.0

func (k Kind) String() string

type Mapper

type Mapper struct {
	// contains filtered or unexported fields
}

func NewMapper

func NewMapper(out *Context) *Mapper

func (*Mapper) Enter

func (m *Mapper) Enter(id int, ext Type) (Type, error)

func (*Mapper) EnterType

func (m *Mapper) EnterType(id int, typ Type)

func (*Mapper) Lookup

func (m *Mapper) Lookup(id int) Type

Lookup tranlates Zed types by type ID from one context to another. The first context is implied by the argument to Lookup() and the output type context is explicitly determined by the argument to NewMapper(). If a binding has not yet been entered, nil is returned and Enter() should be called to create the binding. There is a race here when two threads attempt to update the same ID, but it is safe because the outputContext will return the same the pointer so the second update does not change anything.

type MapperLookupCache added in v1.1.0

type MapperLookupCache struct {
	// contains filtered or unexported fields
}

MapperLookupCache wraps a Mapper with an unsynchronized cache for its Lookup method. Cache hits incur none of the synchronization overhead of Mapper.Lookup.

func (*MapperLookupCache) Lookup added in v1.1.0

func (m *MapperLookupCache) Lookup(id int) Type

func (*MapperLookupCache) Reset added in v1.1.0

func (m *MapperLookupCache) Reset(mapper *Mapper)

type RecordBuilder added in v1.4.0

type RecordBuilder struct {
	// contains filtered or unexported fields
}

func NewRecordBuilder added in v1.4.0

func NewRecordBuilder(zctx *Context, fields field.List) (*RecordBuilder, error)

NewRecordBuilder constructs the zcode.Bytes representation for records built from an array of input field selectors expressed as field.Path. Append should be called to enter field values in the left to right order of the provided fields and Encode is called to retrieve the nested zcode.Bytes value. Reset should be called before encoding the next record.

func (*RecordBuilder) Append added in v1.4.0

func (r *RecordBuilder) Append(leaf []byte)

func (*RecordBuilder) Encode added in v1.4.0

func (r *RecordBuilder) Encode() (zcode.Bytes, error)

func (*RecordBuilder) Reset added in v1.4.0

func (r *RecordBuilder) Reset()

func (*RecordBuilder) Type added in v1.5.0

func (r *RecordBuilder) Type(types []Type) *TypeRecord

A RecordBuilder understands the shape of the field.List from which it was created (i.e., which fields are inside nested records) but not the types. Type takes types for the individual fields and constructs a TypeRecord reflecting the fully typed structure.

type Type

type Type interface {
	// ID returns a unique (per Context) identifier that
	// represents this type.  For a named type, this identifier
	// represents the underlying type and not the named type itself.
	// Callers that care about the underlying type of a Value for
	// example should prefer to use this instead of using a Go
	// type assertion on a Type instance.
	ID() int
	Kind() Kind
}

A Type is an interface presented by a zeek type. Types can be used to infer type compatibility and create new values of the underlying type.

func InnerType

func InnerType(typ Type) Type

InnerType returns the element type for the underlying set or array type or nil if the underlying type is not a set or array.

func LookupPrimitive

func LookupPrimitive(name string) Type

func LookupPrimitiveByID

func LookupPrimitiveByID(id int) (Type, error)

func TypeUnder added in v1.0.0

func TypeUnder(typ Type) Type

func UniqueTypes added in v1.0.0

func UniqueTypes(types []Type) []Type

UniqueTypes returns the set of unique Types in types in sorted order. types will be sorted and deduplicated in place.

type TypeArray

type TypeArray struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeArray

func NewTypeArray(id int, typ Type) *TypeArray

func (*TypeArray) ID

func (t *TypeArray) ID() int

func (*TypeArray) Kind added in v1.0.0

func (t *TypeArray) Kind() Kind

type TypeEnum

type TypeEnum struct {
	Symbols []string
	// contains filtered or unexported fields
}

func NewTypeEnum

func NewTypeEnum(id int, symbols []string) *TypeEnum

func (*TypeEnum) ID

func (t *TypeEnum) ID() int

func (*TypeEnum) Kind added in v1.0.0

func (t *TypeEnum) Kind() Kind

func (*TypeEnum) Lookup

func (t *TypeEnum) Lookup(symbol string) int

func (*TypeEnum) Symbol

func (t *TypeEnum) Symbol(index int) (string, error)

type TypeError

type TypeError struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeError added in v1.0.0

func NewTypeError(id int, typ Type) *TypeError

func (*TypeError) ID added in v1.0.0

func (t *TypeError) ID() int

func (*TypeError) IsMissing added in v1.0.0

func (t *TypeError) IsMissing(zv zcode.Bytes) bool

func (*TypeError) IsQuiet added in v1.0.0

func (t *TypeError) IsQuiet(zv zcode.Bytes) bool

func (*TypeError) Kind added in v1.0.0

func (t *TypeError) Kind() Kind

type TypeMap

type TypeMap struct {
	KeyType Type
	ValType Type
	// contains filtered or unexported fields
}

func NewTypeMap

func NewTypeMap(id int, keyType, valType Type) *TypeMap

func (*TypeMap) Decode

func (t *TypeMap) Decode(zv zcode.Bytes) (Value, Value, error)

func (*TypeMap) ID

func (t *TypeMap) ID() int

func (*TypeMap) Kind added in v1.0.0

func (t *TypeMap) Kind() Kind

type TypeNamed added in v1.0.0

type TypeNamed struct {
	Name string
	Type Type
	// contains filtered or unexported fields
}

func NewTypeNamed added in v1.0.0

func NewTypeNamed(id int, name string, typ Type) *TypeNamed

func (*TypeNamed) ID added in v1.0.0

func (t *TypeNamed) ID() int

func (*TypeNamed) Kind added in v1.0.0

func (t *TypeNamed) Kind() Kind

func (*TypeNamed) NamedID added in v1.0.0

func (t *TypeNamed) NamedID() int

type TypeOfBool

type TypeOfBool struct{}

func (*TypeOfBool) ID

func (t *TypeOfBool) ID() int

func (*TypeOfBool) Kind added in v1.0.0

func (t *TypeOfBool) Kind() Kind

type TypeOfBytes

type TypeOfBytes struct{}

func (*TypeOfBytes) Format

func (t *TypeOfBytes) Format(zv zcode.Bytes) string

func (*TypeOfBytes) ID

func (t *TypeOfBytes) ID() int

func (*TypeOfBytes) Kind added in v1.0.0

func (t *TypeOfBytes) Kind() Kind

type TypeOfDuration

type TypeOfDuration struct{}

func (*TypeOfDuration) ID

func (t *TypeOfDuration) ID() int

func (*TypeOfDuration) Kind added in v1.0.0

func (t *TypeOfDuration) Kind() Kind

type TypeOfFloat16 added in v1.5.0

type TypeOfFloat16 struct{}

func (*TypeOfFloat16) ID added in v1.5.0

func (t *TypeOfFloat16) ID() int

func (*TypeOfFloat16) Kind added in v1.5.0

func (t *TypeOfFloat16) Kind() Kind

type TypeOfFloat32

type TypeOfFloat32 struct{}

func (*TypeOfFloat32) ID

func (t *TypeOfFloat32) ID() int

func (*TypeOfFloat32) Kind added in v1.0.0

func (t *TypeOfFloat32) Kind() Kind

type TypeOfFloat64

type TypeOfFloat64 struct{}

func (*TypeOfFloat64) ID

func (t *TypeOfFloat64) ID() int

func (*TypeOfFloat64) Kind added in v1.0.0

func (t *TypeOfFloat64) Kind() Kind

type TypeOfIP

type TypeOfIP struct{}

func (*TypeOfIP) ID

func (t *TypeOfIP) ID() int

func (*TypeOfIP) Kind added in v1.0.0

func (t *TypeOfIP) Kind() Kind

type TypeOfInt16

type TypeOfInt16 struct{}

func (*TypeOfInt16) ID

func (t *TypeOfInt16) ID() int

func (*TypeOfInt16) Kind added in v1.0.0

func (t *TypeOfInt16) Kind() Kind

type TypeOfInt32

type TypeOfInt32 struct{}

func (*TypeOfInt32) ID

func (t *TypeOfInt32) ID() int

func (*TypeOfInt32) Kind added in v1.0.0

func (t *TypeOfInt32) Kind() Kind

type TypeOfInt64

type TypeOfInt64 struct{}

func (*TypeOfInt64) ID

func (t *TypeOfInt64) ID() int

func (*TypeOfInt64) Kind added in v1.0.0

func (t *TypeOfInt64) Kind() Kind

type TypeOfInt8

type TypeOfInt8 struct{}

func (*TypeOfInt8) ID

func (t *TypeOfInt8) ID() int

func (*TypeOfInt8) Kind added in v1.0.0

func (t *TypeOfInt8) Kind() Kind

type TypeOfNet

type TypeOfNet struct{}

func (*TypeOfNet) ID

func (t *TypeOfNet) ID() int

func (*TypeOfNet) Kind added in v1.0.0

func (t *TypeOfNet) Kind() Kind

type TypeOfNull

type TypeOfNull struct{}

func (*TypeOfNull) ID

func (t *TypeOfNull) ID() int

func (*TypeOfNull) Kind added in v1.0.0

func (t *TypeOfNull) Kind() Kind

type TypeOfString

type TypeOfString struct{}

func (*TypeOfString) ID

func (t *TypeOfString) ID() int

func (*TypeOfString) Kind added in v1.0.0

func (t *TypeOfString) Kind() Kind

type TypeOfTime

type TypeOfTime struct{}

func (*TypeOfTime) ID

func (t *TypeOfTime) ID() int

func (*TypeOfTime) Kind added in v1.0.0

func (t *TypeOfTime) Kind() Kind

type TypeOfType

type TypeOfType struct{}

func (*TypeOfType) ID

func (t *TypeOfType) ID() int

func (*TypeOfType) Kind added in v1.0.0

func (t *TypeOfType) Kind() Kind

type TypeOfUint16

type TypeOfUint16 struct{}

func (*TypeOfUint16) ID

func (t *TypeOfUint16) ID() int

func (*TypeOfUint16) Kind added in v1.0.0

func (t *TypeOfUint16) Kind() Kind

type TypeOfUint32

type TypeOfUint32 struct{}

func (*TypeOfUint32) ID

func (t *TypeOfUint32) ID() int

func (*TypeOfUint32) Kind added in v1.0.0

func (t *TypeOfUint32) Kind() Kind

type TypeOfUint64

type TypeOfUint64 struct{}

func (*TypeOfUint64) ID

func (t *TypeOfUint64) ID() int

func (*TypeOfUint64) Kind added in v1.0.0

func (t *TypeOfUint64) Kind() Kind

type TypeOfUint8

type TypeOfUint8 struct{}

func (*TypeOfUint8) ID

func (t *TypeOfUint8) ID() int

func (*TypeOfUint8) Kind added in v1.0.0

func (t *TypeOfUint8) Kind() Kind

type TypeRecord

type TypeRecord struct {
	Fields []Field
	LUT    map[string]int
	// contains filtered or unexported fields
}

func NewTypeRecord

func NewTypeRecord(id int, fields []Field) *TypeRecord

func TypeRecordOf

func TypeRecordOf(typ Type) *TypeRecord

func (*TypeRecord) HasField

func (t *TypeRecord) HasField(field string) bool

func (*TypeRecord) ID

func (t *TypeRecord) ID() int

func (*TypeRecord) IndexOfField added in v1.9.0

func (t *TypeRecord) IndexOfField(field string) (int, bool)

func (*TypeRecord) Kind added in v1.0.0

func (t *TypeRecord) Kind() Kind

func (*TypeRecord) TypeOfField

func (t *TypeRecord) TypeOfField(field string) (Type, bool)

type TypeSet

type TypeSet struct {
	Type Type
	// contains filtered or unexported fields
}

func NewTypeSet

func NewTypeSet(id int, typ Type) *TypeSet

func (*TypeSet) ID

func (t *TypeSet) ID() int

func (*TypeSet) Kind added in v1.0.0

func (t *TypeSet) Kind() Kind

type TypeUnion

type TypeUnion struct {
	Types []Type
	LUT   map[Type]int
	// contains filtered or unexported fields
}

func NewTypeUnion

func NewTypeUnion(id int, types []Type) *TypeUnion

func (*TypeUnion) ID

func (t *TypeUnion) ID() int

func (*TypeUnion) Kind added in v1.0.0

func (t *TypeUnion) Kind() Kind

func (*TypeUnion) TagOf added in v1.3.0

func (t *TypeUnion) TagOf(typ Type) int

TagOf returns the tag for typ in the union. If no type exists -1 is returned.

func (*TypeUnion) Type

func (t *TypeUnion) Type(tag int) (Type, error)

Type returns the type corresponding to tag.

func (*TypeUnion) Untag added in v1.3.0

func (t *TypeUnion) Untag(bytes zcode.Bytes) (Type, zcode.Bytes)

Untag takes bytes of the reciever's type and returns the underlying value as its type and bytes by removing the tag and determining that tag's type from the union. Untag panics if the tag is invalid.

type TypeVectorTable

type TypeVectorTable struct {
	// contains filtered or unexported fields
}

func NewTypeVectorTable

func NewTypeVectorTable() *TypeVectorTable

func (*TypeVectorTable) Length

func (t *TypeVectorTable) Length() int

func (*TypeVectorTable) Lookup

func (t *TypeVectorTable) Lookup(types []Type) int

func (*TypeVectorTable) LookupByValues

func (t *TypeVectorTable) LookupByValues(vals []Value) int

func (*TypeVectorTable) Types

func (t *TypeVectorTable) Types(id int) []Type

type Value

type Value struct {
	Type Type
	// contains filtered or unexported fields
}

func NewBool

func NewBool(b bool) *Value

func NewBytes

func NewBytes(b []byte) *Value

func NewDuration

func NewDuration(d nano.Duration) *Value

func NewFloat added in v1.9.0

func NewFloat(t Type, f float64) *Value

func NewFloat16 added in v1.5.0

func NewFloat16(f float32) *Value

func NewFloat32

func NewFloat32(f float32) *Value

func NewFloat64

func NewFloat64(f float64) *Value

func NewIP

func NewIP(a netip.Addr) *Value

func NewInt added in v1.9.0

func NewInt(t Type, i int64) *Value

func NewInt16 added in v1.5.0

func NewInt16(i int16) *Value

func NewInt32 added in v1.5.0

func NewInt32(i int32) *Value

func NewInt64 added in v1.5.0

func NewInt64(i int64) *Value

func NewInt8 added in v1.5.0

func NewInt8(i int8) *Value

func NewNet

func NewNet(p netip.Prefix) *Value

func NewString

func NewString(s string) *Value

func NewTime

func NewTime(ts nano.Ts) *Value

func NewTypeValue

func NewTypeValue(t Type) *Value

func NewUint added in v1.9.0

func NewUint(t Type, u uint64) *Value

func NewUint16 added in v1.5.0

func NewUint16(u uint16) *Value

func NewUint32 added in v1.5.0

func NewUint32(u uint32) *Value

func NewUint64

func NewUint64(u uint64) *Value

func NewUint8 added in v1.5.0

func NewUint8(u uint8) *Value

func NewValue added in v0.32.0

func NewValue(zt Type, zb zcode.Bytes) *Value

func (*Value) ArrayIndex

func (v *Value) ArrayIndex(idx int64) (Value, error)

If the passed-in element is an array, attempt to get the idx'th element, and return its type and raw representation. Returns an error if the passed-in element is not an array or if idx is outside the array bounds.

func (*Value) AsBool added in v1.0.0

func (v *Value) AsBool() bool

AsBool returns v's underlying value. It returns false if v is nil or v's underlying type is not TypeBool.

func (*Value) AsInt added in v1.0.0

func (v *Value) AsInt() int64

func (*Value) AsString added in v1.0.0

func (v *Value) AsString() string

func (*Value) AsTime added in v1.0.0

func (v *Value) AsTime() nano.Ts

func (*Value) Bool added in v1.9.0

func (v *Value) Bool() bool

Bool returns v's underlying value. It panics if v's underlying type is not TypeBool.

func (*Value) Bytes

func (v *Value) Bytes() zcode.Bytes

Bytes returns v's ZNG representation.

func (*Value) ContainerLength

func (v *Value) ContainerLength() (int, error)

func (*Value) Copy

func (v *Value) Copy() *Value

Copy returns a copy of v that shares no storage.

func (*Value) CopyFrom added in v1.0.0

func (v *Value) CopyFrom(from *Value)

CopyFrom copies from into v, reusing v's storage if possible.

func (*Value) Deref added in v0.32.0

func (v *Value) Deref(field string) *Value

func (*Value) DerefByColumn added in v1.0.0

func (v *Value) DerefByColumn(col int) *Value

func (*Value) DerefPath added in v1.0.0

func (v *Value) DerefPath(path field.Path) *Value

func (*Value) Elements

func (v *Value) Elements() ([]Value, error)

Elements returns an array of Values for the given container type. Returns an error if the element is not an array or set.

func (*Value) Encode

func (v *Value) Encode(dst zcode.Bytes) zcode.Bytes

Encode appends the ZNG representation of this value to the passed in argument and returns the resulting zcode.Bytes (which may or may not be the same underlying buffer, as with append(), depending on its capacity)

func (*Value) Equal

func (v *Value) Equal(p Value) bool

Equal reports whether p and v have the same type and the same ZNG representation.

func (*Value) Fields added in v1.5.0

func (r *Value) Fields() []Field

func (*Value) Float added in v1.9.0

func (v *Value) Float() float64

Float returns v's underlying value. It panics if v's underlying type is not TypeFloat16, TypeFloat32, or TypeFloat64.

func (*Value) HasField added in v0.32.0

func (r *Value) HasField(field string) bool

func (*Value) IndexOfField added in v1.9.0

func (v *Value) IndexOfField(field string) (int, bool)

func (*Value) Int added in v1.9.0

func (v *Value) Int() int64

Int returns v's underlying value. It panics if v's underlying type is not TypeInt8, TypeInt16, TypeInt32, TypeInt64, TypeDuration, or TypeTime.

func (*Value) IsContainer

func (v *Value) IsContainer() bool

func (*Value) IsError

func (v *Value) IsError() bool

func (*Value) IsMissing

func (v *Value) IsMissing() bool

func (*Value) IsNull added in v1.0.0

func (v *Value) IsNull() bool

IsNull returns true if and only if v is a null value of any type.

func (*Value) IsQuiet added in v1.0.0

func (v *Value) IsQuiet() bool

func (*Value) IsString added in v1.0.0

func (v *Value) IsString() bool

func (*Value) Iter

func (v *Value) Iter() zcode.Iter

func (*Value) MissingAsNull added in v1.0.0

func (v *Value) MissingAsNull() *Value

func (*Value) String

func (v *Value) String() string

String implements fmt.Stringer.String. It should only be used for logs, debugging, etc. Any caller that requires a specific output format should use FormatAs() instead.

func (*Value) Uint added in v1.9.0

func (v *Value) Uint() uint64

Uint returns v's underlying value. It panics if v's underlying type is not TypeUint8, TypeUint16, TypeUint32, or TypeUint64.

func (*Value) Under added in v1.3.0

func (v *Value) Under() *Value

func (*Value) Validate added in v1.4.0

func (v *Value) Validate() (err error)

Validate checks that v.Bytes is structurally consistent with v.Type. It does not check that the actual leaf values when parsed are type compatible with the leaf types.

func (*Value) Walk added in v0.32.0

func (r *Value) Walk(rv Visitor) error

Walk traverses a value in depth-first order, calling a Visitor on the way.

type Visitor

type Visitor func(typ Type, body zcode.Bytes) error

A Visitor is called for each value in a record encountered by Walk. If the visitor returns an error, the walk stops and that error will be returned to the caller of Walk(). The sole exception is when the visitor returns the special value SkipContainer.

Directories

Path Synopsis
api
cli
zq
cmd
zc
zed
zq
ast
Package ast declares the types used to represent syntax trees for Zed queries.
Package ast declares the types used to represent syntax trees for Zed queries.
Package index provides an API for creating, merging, indexing, and querying Zed indexes.
Package index provides an API for creating, merging, indexing, and querying Zed indexes.
api
Package mdtest finds example shell commands in Markdown files and runs them, checking for expected output and exit status.
Package mdtest finds example shell commands in Markdown files and runs them, checking for expected output and exit status.
pkg
bufwriter
Package bufwriter provides a wrapper for a io.WriteCloser that uses buffered output via a bufio.Writer and calls Flush on close.
Package bufwriter provides a wrapper for a io.WriteCloser that uses buffered output via a bufio.Writer and calls Flush on close.
byteconv
Package byteconv implements conversions from byte slice representations of various data types.
Package byteconv implements conversions from byte slice representations of various data types.
charm
Package charm is minimilast CLI framework inspired by cobra and urfave/cli.
Package charm is minimilast CLI framework inspired by cobra and urfave/cli.
colw
Package colw lays out columns for display of a list when you don't know ahead of time how many columns should exist.
Package colw lays out columns for display of a list when you don't know ahead of time how many columns should exist.
fs
repl
Package repl is a simple read-eval-print loop.
Package repl is a simple read-eval-print loop.
storage/cache
Package cache contains facilities for caching immutable files, typically for a cloud object store.
Package cache contains facilities for caching immutable files, typically for a cloud object store.
storage/mock
Package mock is a generated GoMock package.
Package mock is a generated GoMock package.
unpack
Package unpack provides a means to unmarshal Go values that have embedded interface values.
Package unpack provides a means to unmarshal Go values that have embedded interface values.
op
op/combine
A combine proc merges multiple upstream inputs into one output.
A combine proc merges multiple upstream inputs into one output.
srverr
Package zqe provides a mechanism to create or wrap errors with information that will aid in reporting them to users and returning them to api callers.
Package zqe provides a mechanism to create or wrap errors with information that will aid in reporting them to users and returning them to api callers.
vng
Package vng implements the reading and writing of VNG storage objects to and from any Zed format.
Package vng implements the reading and writing of VNG storage objects to and from any Zed format.
vector
Package vector implements the organization of Zed data on storage as vectors in a VNG vector storage object.
Package vector implements the organization of Zed data on storage as vectors in a VNG vector storage object.
Package zcode implements serialization and deserialzation for ZNG values.
Package zcode implements serialization and deserialzation for ZNG values.
zio
zngio
Package zngio provides an API for reading and writing zng values and directives in binary zng format.
Package zngio provides an API for reading and writing zng values and directives in binary zng format.
Package zson provides fundamental interfaces to the ZSON data format comprising Reader, Writer, Parser, and so forth.
Package zson provides fundamental interfaces to the ZSON data format comprising Reader, Writer, Parser, and so forth.
Package ztest runs formulaic tests ("ztests") that can be (1) run in-process with the compiled-ini zq code base, (2) run as a sub-process using the zq executable build artifact, or (3) run as a bash script running a sequence of arbitrary shell commands invoking any of the build artifacts.
Package ztest runs formulaic tests ("ztests") that can be (1) run in-process with the compiled-ini zq code base, (2) run as a sub-process using the zq executable build artifact, or (3) run as a bash script running a sequence of arbitrary shell commands invoking any of the build artifacts.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL