ion

package
v0.0.0-...-86e9f11 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 7, 2024 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package ion implements a subset of the Amazon ion binary format: https://amzn.github.io/ion-docs/

Index

Constants

This section is empty.

Variables

View Source 
var (
	False = Datum{/* contains filtered or unexported fields */}
	True  = Datum{/* contains filtered or unexported fields */}
)
View Source 
var Empty = Datum{}

Empty is the zero value of a Datum.

View Source 
var (
	ErrTooLarge = errors.New("ion: object size exceeds max size")
)
View Source 
var ErrUnexpectedEnd = errors.New("unexpected end of list")

ErrUnexpectedEnd is returned if Iterator.Next is called after the list has been exhausted.

View Source 
var Null = Datum{/* contains filtered or unexported fields */}

Null is the untyped null datum.

View Source 
var Stop = errors.New("stop early")

Stop can be returned by the function passed to List.Each and Struct.Each to stop iterating and return early.

Functions

func Contents 

func Contents(msg []byte) ([]byte, []byte)

Contents parses the TLV descriptor at the beginning of 'msg' and returns the bytes that correspond to the non-descriptor bytes of the object, plus the remaining bytes in the buffer as the second return value. The returned []byte will be nil if the encoded object size does not fit into 'msg'. (Note that a returned slice that is zero-length but non-nil means something different than a nil slice.)

func Equal 

func Equal(a, b Datum) bool

Equal returns whether a and b are semantically equivalent.

func HeaderSizeOf 

func HeaderSizeOf(msg []byte) int

HeaderSizeOf returns the size of the next ION object's header. The function counts TLV byte and the size of the optional Length field.

NOTE: This function only counts bytes, it doesn't return the real size of the value.

func IsBVM 

func IsBVM(buf []byte) bool

IsBVM returns whether or not the next 4 bytes of the message are a 4-byte ion BVM marker.

func Marshal 

func Marshal(st *Symtab, dst *Buffer, src any) error

Marshal encodes src into dst, updating the symbol table as necessary for new symbols that are introduced as part of encoding.

func MinimumID 

func MinimumID(str string) int

MinimumID returns the lowest ID that a string could be symbolized as.

System symbols have IDs less than 10; all other symbols have and ID of at least 10.

func NopPadding 

func NopPadding(dst []byte, padsize int) (int, int)

NopPadding writes a header for Ion NOP operation for given padsize. It returns the size of header and the number of pad bytes that have to be written after the header.

FIXME: it's not possible to produce exact padding for all possible padsize. The procedure never returns values that would exceed the required padding.

func ReadBool 

func ReadBool(msg []byte) (bool, []byte, error)

ReadBool reads a boolean value and returns it along with the subsequent message bytes

func ReadBytes 

func ReadBytes(msg []byte) ([]byte, []byte, error)

ReadBytes reads an ion blob from message. The returned slice does not alias msg. See also: ReadBytesShared.

func ReadBytesShared 

func ReadBytesShared(msg []byte) ([]byte, []byte, error)

ReadBytesShared read a []byte (as an ion 'blob') and returns the blob and the subsequent message bytes. Note that the returned []byte aliases the input message, so the caller must copy those bytes into a new buffer if the original buffer is expected to be clobbered.

func ReadCoerceFloat64 

func ReadCoerceFloat64(msg []byte) (float64, []byte, error)

func ReadFloat32 

func ReadFloat32(msg []byte) (float32, []byte, error)

ReadFloat32 reads an ion float as a float32 and returns the value and the subsequent message bytes.

func ReadFloat64 

func ReadFloat64(msg []byte) (float64, []byte, error)

ReadFloat64 reads an ion float as a float64 and returns the value and the subsequent message bytes.

func ReadInt 

func ReadInt(msg []byte) (int64, []byte, error)

ReadInt reads an ion integer as an int64 and returns the subsequent message bytes

func ReadIntMagnitude 

func ReadIntMagnitude(msg []byte) (uint64, []byte, error)

ReadIntMagnitude reads magnitude of an integer (either signed or unsigned) and returns the subsequent message bytes

func ReadString 

func ReadString(msg []byte) (string, []byte, error)

ReadString reads a string from 'msg' and returns the string and the subsequent message bytes.

func ReadStringShared 

func ReadStringShared(msg []byte) ([]byte, []byte, error)

ReadStringShared reads a string from 'msg' and returns the string and the subsequent message bytes. The returned slice containing the string contents aliases the input slice.

func ReadTime 

func ReadTime(msg []byte) (date.Time, []byte, error)

ReadTime reads a timestamp object and returns the subsequent message bytes.

func ReadUint 

func ReadUint(msg []byte) (uint64, []byte, error)

ReadUint reads an ion integer as a uint64 and returns the subsequent message bytes

func SizeOf 

func SizeOf(msg []byte) int

SizeOf returns the size of the next ion object, including the beginning TLV descriptor bytes.

The return value of SizeOf is unspecified when msg is not a valid ion object.

func ToJSON 

func ToJSON(w io.Writer, r *bufio.Reader) (int, error)

ToJSON reads a stream of ion objects from 'r' and writes them to 'w'. Each top-level object in the stream of objects is written on its own line. (See also: jsonlines.org, ndjson.org)

Ion structures are written as json objects, lists and sexps are written as arrays, symbols, strings and timestamps are written as strings, blobs and clobs are written as base64-encoded strings, numbers are written as numbers (using as many characters as are necessary to preserve precision upon decoding the numbers), and null objects are written as JSON nulls.

Symbol tables present in the stream of objects are decoded and used to print symbols and structure fields. Per the ion binary specification, a BVM marker flushes the current symbol table. (Annotation objects and padding objects do not produce any JSON output.)

ToJSON returns the number of bytes written to w and the first error encountered (if any).

func Unmarshal 

func Unmarshal(st *Symtab, data []byte, v any) ([]byte, error)

Unmarshal unmarshals data from a raw slice into the value v using the provided symbol table.

func UnpackList 

func UnpackList(body []byte, fn func([]byte) error) (rest []byte, err error)

UnpackList calls fn for each item in a list, returning the remaining bytes.

func UnpackStruct 

func UnpackStruct(st *Symtab, body []byte, fn func(string, []byte) error) (rest []byte, err error)

UnpackStruct calls fn for each field in a struct, returning the remaining bytes.

func UnpackStructBody 

func UnpackStructBody(st *Symtab, body []byte, fn func(string, []byte) error) (rest []byte, err error)

UnpackStructBody calls fn for each field in a struct, assuming that `body` is an already extracted record content, without an Ion header (the TLV byte and object size).

func UnpackTyped 

func UnpackTyped[T FieldSetter](d Datum, fn func(typ string) (T, bool)) (T, error)

UnpackTyped iterates the fields in a struct to find a string field named "type" which is passed to fn to resolve to a concrete type. The other fields are passed to SetField on the returned object.

fn should return true to indicate that the type was resolved, and false to indicate that the type is not supported.

If d is not a struct or the "type" field is not present, this returns an error.

func UnsafeAppendTag 

func UnsafeAppendTag(dst []byte, tag Type, size uint) []byte

UnsafeAppendTag appends a type+length tag of the given type and size to dst and returns the extended buffer.

func UnsafeWriteTag 

func UnsafeWriteTag(dst []byte, tag Type, size uint) int

UnsafeWriteTag writes the header of an Ion object having size. Returns the number of stored bytes

It's required that dst has enough room for the encoded header.

func UnsafeWriteUVarint 

func UnsafeWriteUVarint(dst []byte, uv uint) int

UnsafeWriteUVarint encodes uv as a uvarint number. Returns the number of stored bytes.

It's required that dst has enough room for the encoded number (i.e., len(buf) >= Uvsize(uv)).

func Uvsize 

func Uvsize(value uint) int

Uvsize returns the encoded size of value as a uvarint

Types

type Bag 

type Bag struct {
	// contains filtered or unexported fields
}

Bag is a (possibly empty) sequence of ion datums. A bag is stored efficiently in memory so as to reduce the CPU and memory footprint of the constituent data elements.

func (*Bag) Add 

func (b *Bag) Add(st *Symtab, raw []byte) error

Add adds zero or more raw ion datums from a buffer and an associated symbol table.

func (*Bag) AddDatum 

func (b *Bag) AddDatum(d Datum)

AddDatum adds a single datum to the bag.

func (*Bag) AddList 

func (b *Bag) AddList(d Datum) error

AddList adds the contents of a list to the bag. If d is not a list, this returns an error.

func (*Bag) Append 

func (b *Bag) Append(src *Bag)

Append appends the contents of src to b.

func (*Bag) Clone 

func (b *Bag) Clone() Bag

Clone creates a deep copy of the Bag.

func (*Bag) Each 

func (b *Bag) Each(fn func(d Datum) bool)

Each iterates each item in the bag and calls fn() on it. If fn returns false, then Each returns early.

func (*Bag) EachPair 

func (b *Bag) EachPair(x *Bag, fn func(b, x Datum) bool)

EachPair iterates b and x up to min(b.Len(), x.Len()) and returns the corresponding values in each bag at each position.

func (*Bag) Encode 

func (b *Bag) Encode(dst *Buffer, st *Symtab)

Encode encodes the contents of the bag to dst using the symbol table st.

func (*Bag) Equals 

func (b *Bag) Equals(o *Bag) bool

Equals compares two bags and returns true if they are equivalent.

Note that Equals is sensitive to the order of items in each Bag.

func (*Bag) Len 

func (b *Bag) Len() int

Len returns the number of items in the bag.

func (*Bag) Raw 

func (b *Bag) Raw() []byte

Raw returns the raw encoded datums. The caller must not alter the contents of the returned slice.

func (*Bag) Reset 

func (b *Bag) Reset()

func (*Bag) Size 

func (b *Bag) Size() int

Size returns the *approximate* size of the bag in memory.

func (*Bag) Transcoder 

func (b *Bag) Transcoder(st *Symtab) func(dst *Buffer, src Datum)

Transcoder returns a function that can be used to efficiently transcode datums from within the bag to a different symbol table. The returned function is only valid for use with Datum objects returned from b.Each.

func (*Bag) Writer 

func (b *Bag) Writer() io.Writer

Writer returns an io.Writer that can be used to write data directly into the bag.

type Buffer 

type Buffer struct {
	// contains filtered or unexported fields
}

Buffer buffers ion objects.

The contents of Buffer can be inspected directly with Buffer.Bytes() or written to an io.Writer with Buffer.WriteTo.

func (*Buffer) BeginAnnotation 

func (b *Buffer) BeginAnnotation(labels int)

BeginAnnotation begins an annotation object. 'labels' should indicate the number annotation fields before the wrapped object, and must be greater than zero.

func (*Buffer) BeginField 

func (b *Buffer) BeginField(sym Symbol)

BeginField begins a field of a structure or a label of an annotation. BeginField will panic if the buffer is not in an appropriate structure field context

func (*Buffer) BeginList 

func (b *Buffer) BeginList(hint int)

BeginList begins a list object. Subsequent calls to the Buffer.Write* methods will write list elements until Buffer.EndList is called.

func (*Buffer) BeginStruct 

func (b *Buffer) BeginStruct(hint int)

BeginStruct begins a structure. Fields of the structure should be written with paired calls to BeginField and one of the Write* methods, followed by Buffer.EndStruct.

func (*Buffer) Bytes 

func (b *Buffer) Bytes() []byte

Bytes returns the current contents of the buffer.

func (*Buffer) EndAnnotation 

func (b *Buffer) EndAnnotation()

EndAnnotation ends an annotation object.

func (*Buffer) EndList 

func (b *Buffer) EndList()

EndList ends a list object.

If EndList is not paried with a corresponding BeginList call, it will panic.

func (*Buffer) EndStruct 

func (b *Buffer) EndStruct()

EndStruct ends a structure.

If EndStruct is not paired with a corresponding BeginStruct call, it will panic.

func (*Buffer) Ok 

func (b *Buffer) Ok() bool

Ok returns false if there are any open calls to BeginStruct or BeginList that have not been paired with EndStruct or EndList, respectively.

func (*Buffer) Reset 

func (b *Buffer) Reset()

Reset resets a buffer to its initial state.

func (*Buffer) Set 

func (b *Buffer) Set(p []byte)

Set sets the buffer used by 'b' and resets the state of the buffer. Subsequent calls to Write* functions on 'b' will append to the given buffer.

func (*Buffer) Size 

func (b *Buffer) Size() int

Size returns the number of bytes in the buffer.

func (*Buffer) StartChunk 

func (b *Buffer) StartChunk(symtab *Symtab)

StartChunk writes BVM marker and symtab.

func (*Buffer) UnsafeAppend 

func (b *Buffer) UnsafeAppend(buf []byte)

UnsafeAppend appends arbitrary data to the buffer. If the buffer is currently in List, Struct, or Annotation context, then the contents of buf should be exactly one ion datum.

func (*Buffer) UnsafeAppendFields 

func (b *Buffer) UnsafeAppendFields(buf []byte)

UnsafeAppendFields appends an encoded field list as an ion structure. The data must be zero or more encoded (uvarint, field) pairs with the fields sorted in ascending symbol ID order.

func (*Buffer) WriteBlob 

func (b *Buffer) WriteBlob(p []byte)

WriteBlob writes a []byte as an ion 'blob' to the buffer.

func (*Buffer) WriteBool 

func (b *Buffer) WriteBool(n bool)

WriteBool writes a bool into the buffer

func (*Buffer) WriteCanonicalFloat 

func (b *Buffer) WriteCanonicalFloat(f float64)

func (*Buffer) WriteFloat32 

func (b *Buffer) WriteFloat32(f float32)

WriteFloat32 writes an ion float32 to the buffer

func (*Buffer) WriteFloat64 

func (b *Buffer) WriteFloat64(f float64)

WriteFloat64 writes an ion float64 to the buffer

func (*Buffer) WriteInt 

func (b *Buffer) WriteInt(i int64)

WriteInt writes an integer to the buffer.

func (*Buffer) WriteList 

func (b *Buffer) WriteList(st *Symtab, items []Datum)

func (*Buffer) WriteNull 

func (b *Buffer) WriteNull()

WriteNull writes an ion NULL value into the buffer

func (*Buffer) WriteString 

func (b *Buffer) WriteString(s string)

WriteString writes a string as an ion string into a Buffer

func (*Buffer) WriteStringBytes 

func (b *Buffer) WriteStringBytes(s []byte)

WriteStringBytes works identically to WriteString, but it uses a []byte as the string contents rather than a string.

func (*Buffer) WriteStruct 

func (b *Buffer) WriteStruct(st *Symtab, f []Field)

func (*Buffer) WriteSymbol 

func (b *Buffer) WriteSymbol(s Symbol)

WriteSymbol writes an ion symbol value to the buffer.

func (*Buffer) WriteTime 

func (b *Buffer) WriteTime(t date.Time)

WriteTime writes a date.Date as an ion timestamp object.

WriteTime only supports microsecond-precision timestamps.

func (*Buffer) WriteTo 

func (b *Buffer) WriteTo(w io.Writer) (int64, error)

WriteTo implements io.WriterTo

func (*Buffer) WriteTruncatedTime 

func (b *Buffer) WriteTruncatedTime(t date.Time, trunc TimeTrunc)

WriteTruncatedTime writes a date.Date as an ion timestamp object and lets the caller decide how precise the output is.

func (*Buffer) WriteUint 

func (b *Buffer) WriteUint(u uint64)

WriteUint writes an unsigned integer to the buffer.

type Chunker 

type Chunker struct {
	// Buffer is the current
	// buffered data.
	Buffer
	// Symbols is the current symbol table.
	Symbols Symtab

	// Align is the alignment of output data.
	// (Align should not be modified once the
	// data has begun being committed to the Chunker.)
	Align int
	// RangeAlign is the alignment
	// at which ranges are flushed to W.
	RangeAlign int
	// W is the output io.Writer.
	// All writes to W will have length equal to Align.
	//
	// If W implements Flusher, then Flush is called
	// immediately after ranges are written, which
	// should occur at most once every RangeAlign bytes.
	W io.Writer
	// Ranges stores field ranges for the current
	// chunk.
	Ranges Ranges

	// WalkTimeRanges is the list of time ranges
	// that is automatically scanned during
	// Chunker.Write.
	WalkTimeRanges [][]string
	// contains filtered or unexported fields
}

Chunker is a wrapper for a Buffer and Symtab that allows objects to be written to an output stream on aligned boundaries with padding.

In order to use a Chunker, populate its Align and W fields with the target alignment and destination of the ion output. Then, write objects to Buffer (optionally updating Symbols) and call Commit after each object has been written. After the complete object stream has been written, call Flush to flush any remaining buffer contents.

func (*Chunker) CheckSize 

func (c *Chunker) CheckSize() error

CheckSize checks that the current number of un-commited bytes will fit within c.Align. If the object would not fit, it returns an error immediately.

func (*Chunker) Commit 

func (c *Chunker) Commit() error

Commit commits an object to the state buffer, taking care to flush it if we would exceed the block alignment.

Note that Commit will refuse to commit objects that do not fit in the target output alignment.

Commit should be called after each complete object has been written to a chunker.

func (*Chunker) FastForward 

func (c *Chunker) FastForward(n int)

FastForward changes the initial values for the number of flushed bytes to c.W and the contents of the chunker ranges. c.Symbols should be set to whatever the "current" symbol table is expected to be.

func (*Chunker) Flush 

func (c *Chunker) Flush() error

Flush flushes the output of the chunker, regardless of whether or not the current buffer is approaching the target alignment. (The output to c.W will still be padded to the appropriate alignment.)

Flush must always be preceded by a call to Commit unless zero objects have been written to c.Buffer.

func (*Chunker) ReadFrom 

func (c *Chunker) ReadFrom(r io.Reader, cons []Field) (int64, error)

ReadFrom reads ion from r and re-encodes it into the chunker by reading objects one-at-a-time. If cons is provided, these fields will be added to each structure.

BUGS: ReadFrom only indexes data from the top-level of each structure.

func (*Chunker) Reset 

func (c *Chunker) Reset()

Reset resets c to its initial state. This should only be used between benchmark runs to avoid allocation overhead.

func (*Chunker) Set 

func (c *Chunker) Set(b []byte)

Set sets the buffer used by c to b and resets c to its initial state. This should only be used between benchmark runs to avoid allocation overhead.

func (*Chunker) SetTimeRange 

func (c *Chunker) SetTimeRange(p []string, min, max date.Time)

SetTimeRange clobbers the currently-stored time range values

func (*Chunker) Write 

func (c *Chunker) Write(block []byte) (int, error)

Write writes a block of ion data from a stream. If write does not begin with a BVM and/or symbol table, then previous calls to Write must have already set the symbol table. (The output stream of Chunker is compatible with Write.)

type Datum 

type Datum struct {
	// contains filtered or unexported fields
}

Datum represents any Ion datum.

The Marshal and Unmarshal functions natively understand that Datum can be constructed and re-encoded from any ion value.

A Datum should be a value returned by 

Float, Int, Uint, Struct, List, Bool,
BigInt, Timestamp, Annotation, ..., or ReadDatum.

func Annotation 

func Annotation(st *Symtab, label string, val Datum) Datum

Annotation objects represent ion annotation datums.

func Blob 

func Blob(b []byte) Datum

func Bool 

func Bool(b bool) Datum

func Float 

func Float(f float64) Datum

func FromJSON 

func FromJSON(st *Symtab, d *json.Decoder) (Datum, error)

FromJSON decodes one JSON datum from 'd' and returns it as an ion Datum.

func Int 

func Int(i int64) Datum

func Interned 

func Interned(st *Symtab, s string) Datum

Interned returns a Datum that represents an interned string (a Symbol). Interned is always encoded as an ion symbol.

func NewString 

func NewString(s string) Datum

NewString constructs a new ion Datum from a string.

func ReadDatum 

func ReadDatum(st *Symtab, buf []byte) (Datum, []byte, error)

ReadDatum reads the next datum from buf and returns it. ReadDatum does not return symbol tables directly; instead it unmarshals them into st and continues reading. It may return a nil datum if buf points to a symbol table followed by zero bytes of actual ion data.

Any Symbol datums in buf are translated into Interned datums rather than Symbol datums, as this makes the returned Datum safe to re-encode with a new symbol table.

The returned datum will share memory with buf and so the caller must guarantee that the contents of buf will not be modified until it is no longer needed.

func String 

func String(s string) Datum

func Timestamp 

func Timestamp(t date.Time) Datum

func Uint 

func Uint(u uint64) Datum

func (Datum) Annotation 

func (d Datum) Annotation() (string, Datum, error)

func (Datum) Blob 

func (d Datum) Blob() ([]byte, error)

func (Datum) BlobShared 

func (d Datum) BlobShared() ([]byte, error)

BlobShared returns a []byte aliasing the contents of this Datum and should be copied as necessary to avoid issues that may arise from retaining aliased bytes.

func (Datum) Bool 

func (d Datum) Bool() (bool, error)

func (Datum) Clone 

func (d Datum) Clone() Datum

func (Datum) CloneInto 

func (d Datum) CloneInto(dst *Datum)

CloneInto clobbers the underlying storage for dst with the contents of d.

func (Datum) CoerceFloat 

func (d Datum) CoerceFloat() (float64, error)

func (Datum) DecodeRelated 

func (d Datum) DecodeRelated(b []byte) (Datum, error)

DecodeRelated attempts to decode a datum that was encoded using the same symbol table as this datum, for example when a struct contains a blob holding the compressed encoded form of another composite object.

This method only works with lists and structs. Calling this method on any other data type will result in an error.

func (Datum) Encode 

func (d Datum) Encode(dst *Buffer, st *Symtab)

func (Datum) Equal 

func (d Datum) Equal(x Datum) bool

Equal returns whether d and x are semantically equivalent.

func (Datum) Field 

func (d Datum) Field(name string) Datum

Field returns the value associated with the field with the given name if d is a struct. If d is not a struct or the field is not present, this returns Empty.

func (Datum) Float 

func (d Datum) Float() (float64, error)

func (Datum) Int 

func (d Datum) Int() (int64, error)

func (Datum) IsAnnotation 

func (d Datum) IsAnnotation() bool

func (Datum) IsBlob 

func (d Datum) IsBlob() bool

func (Datum) IsBool 

func (d Datum) IsBool() bool

func (Datum) IsEmpty 

func (d Datum) IsEmpty() bool

func (Datum) IsFloat 

func (d Datum) IsFloat() bool

func (Datum) IsInt 

func (d Datum) IsInt() bool

func (Datum) IsList 

func (d Datum) IsList() bool

func (Datum) IsNull 

func (d Datum) IsNull() bool

func (Datum) IsString 

func (d Datum) IsString() bool

func (Datum) IsStruct 

func (d Datum) IsStruct() bool

func (Datum) IsSymbol 

func (d Datum) IsSymbol() bool

func (Datum) IsTimestamp 

func (d Datum) IsTimestamp() bool

func (Datum) IsUint 

func (d Datum) IsUint() bool

func (Datum) Iterator 

func (d Datum) Iterator() (Iterator, error)

Iterator calls d.List and calls Iterator on the result.

func (Datum) JSON 

func (d Datum) JSON() string

func (Datum) LessImprecise 

func (d Datum) LessImprecise(x Datum) bool

LessImprecise compares the raw Ion bytes.

This method does not order correctly equal values having different binary representation. For instance a string can be saved literally, as a sequence of UTF-8 bytes, or be a symbol, that is a reference to the symbol table.

func (Datum) List 

func (d Datum) List() (List, error)

func (Datum) Null 

func (d Datum) Null() error

func (Datum) Raw 

func (d Datum) Raw() []byte

Raw returns the raw byte slice underlying d. The returned slice aliases the contents of d, so care should be taken when retaining or modifying the returned slice.

func (Datum) String 

func (d Datum) String() (string, error)

func (Datum) StringShared 

func (d Datum) StringShared() ([]byte, error)

StringShared returns a []byte aliasing the contents of this Datum and should be copied as necessary to avoid issues that may arise from retaining aliased bytes.

Unlike String, this method will not work with a symbol datum.

func (Datum) Struct 

func (d Datum) Struct() (Struct, error)

func (Datum) Symbol 

func (d Datum) Symbol() (Symbol, error)

func (Datum) Timestamp 

func (d Datum) Timestamp() (date.Time, error)

func (Datum) Type 

func (d Datum) Type() Type

func (Datum) Uint 

func (d Datum) Uint() (uint64, error)

func (Datum) UnpackList 

func (d Datum) UnpackList(fn func(Datum) error) error

UnpackList calls d.List and calls UnpackList on the result.

func (Datum) UnpackStruct 

func (d Datum) UnpackStruct(fn func(Field) error) error

UnpackStruct calls d.Struct and calls UnpackStruct on the result.

type Decoder 

type Decoder struct {
	// Symtab is the current symbol table.
	// Calls to Decoder.Decode will update
	// the symbol table as symbol table annotations
	// are encountered in the source data stream.
	Symbols Symtab
	// ExtraAnnotations holds additional values
	// that will be Unmarshal()'ed into when the
	// associated annotation label occurs at the top
	// level in the stream. In other words, if the stream
	// contains
	//   label::{foo: "bar"}
	// then the Decoder will look up "label" in ExtraAnnotations
	// and Unmarshal the {foo: "bar"} value into the associated
	// Go value.
	ExtraAnnotations map[string]any
	// contains filtered or unexported fields
}

Decoder is a stateful decoder of streams of ion objects. A Decoder wraps an io.Reader so that a sequence of ion records can be read via Decoder.Decode.

See also: Unmarshal.

func NewDecoder 

func NewDecoder(r io.Reader, max int) *Decoder

NewDecoder constructs a decoder that reads objects from r up to the given maximum size.

func (*Decoder) Decode 

func (d *Decoder) Decode(dst any) error

Decode buffers the next object in the source stream and calls Unmarshal(&d.Symtab, buffer, dst). Decode will return bufio.ErrBufferFull if the source object is larger than the maximum permitted object size for the decoder.

func (*Decoder) MaxSize 

func (d *Decoder) MaxSize() int

MaxSize reports the maximum object size that the Decoder supports unmarshaling via d.Decode.

type Field 

type Field struct {
	Label string
	Datum
	Sym Symbol // symbol, if assigned
}

Field is a structure field in a Struct or Annotation datum

func ReadField 

func ReadField(st *Symtab, body []byte) (Field, []byte, error)

func (Field) Annotation 

func (f Field) Annotation() (string, Datum, error)

func (Field) Blob 

func (f Field) Blob() ([]byte, error)

func (Field) BlobShared 

func (f Field) BlobShared() ([]byte, error)

BlobShared returns a []byte aliasing the contents of this Field and should be copied as necessary to avoid issues that may arise from retaining aliased bytes.

func (Field) Bool 

func (f Field) Bool() (bool, error)

func (Field) Clone 

func (f Field) Clone() Field

func (*Field) Encode 

func (f *Field) Encode(dst *Buffer, st *Symtab)

func (*Field) Equal 

func (f *Field) Equal(f2 *Field) bool

func (Field) Float 

func (f Field) Float() (float64, error)

func (Field) Int 

func (f Field) Int() (int64, error)

func (Field) Iterator 

func (f Field) Iterator() (Iterator, error)

Iterator calls f.List and calls Iterator on the result.

func (Field) List 

func (f Field) List() (List, error)

func (Field) Null 

func (f Field) Null() error

func (Field) String 

func (f Field) String() (string, error)

func (Field) StringShared 

func (f Field) StringShared() ([]byte, error)

StringShared returns a []byte aliasing the contents of this Field and should be copied as necessary to avoid issues that may arise from retaining aliased bytes.

Unlike String, this method will not work with a symbol datum.

func (Field) Struct 

func (f Field) Struct() (Struct, error)

func (Field) Symbol 

func (f Field) Symbol() (Symbol, error)

func (Field) Timestamp 

func (f Field) Timestamp() (date.Time, error)

func (Field) Uint 

func (f Field) Uint() (uint64, error)

func (Field) UnpackList 

func (f Field) UnpackList(fn func(Datum) error) error

UnpackList calls f.List and calls UnpackList on the result.

func (Field) UnpackStruct 

func (f Field) UnpackStruct(fn func(Field) error) error

UnpackStruct calls f.Struct and calls UnpackStruct on the result.

type FieldSetter 

type FieldSetter interface {
	// SetField is called for each structure field.
	SetField(Field) error
}

FieldSetter is an object which can accept fields from an ion struct.

type Flusher 

type Flusher interface {
	Flush() error
}

Flusher is an interface optionally implemented by io.Writers that would like to be notified when ranges have been flushed.

See ion.Chunker.W.

type Iterator 

type Iterator struct {
	// contains filtered or unexported fields
}

An Iterator can be used to iterate over items in a list.

func (*Iterator) Blob 

func (i *Iterator) Blob() ([]byte, error)

func (*Iterator) Bool 

func (i *Iterator) Bool() (bool, error)

func (*Iterator) Done 

func (i *Iterator) Done() bool

Done returns whether the iterator has reached the end of the list. If Done returns false, the next call to Next will not return

func (*Iterator) Float 

func (i *Iterator) Float() (float64, error)

func (*Iterator) Int 

func (i *Iterator) Int() (int64, error)

func (*Iterator) List 

func (i *Iterator) List() (List, error)

func (*Iterator) Next 

func (i *Iterator) Next() (Datum, error)

Next returns the next item in the list. The caller should check Done first to ensure that there are still items in the list.

If Next is called after reaching the the end of the list, this returns ErrUnexpectedEnd. If there is an error decoding the next item in the list, that error is returned.

func (*Iterator) String 

func (i *Iterator) String() (string, error)

func (*Iterator) Struct 

func (i *Iterator) Struct() (Struct, error)

func (*Iterator) Symbol 

func (i *Iterator) Symbol() (Symbol, error)

func (*Iterator) Timestamp 

func (i *Iterator) Timestamp() (date.Time, error)

func (*Iterator) Uint 

func (i *Iterator) Uint() (uint64, error)

type JSONWriter 

type JSONWriter struct {
	// W is the output io.Writer into which
	// the JSON data is written.
	W io.Writer
	// ShowAnnotations causes top-level annotation
	// objects to be displayed.
	//
	// An ion annotation x::y is encoded
	// as a JSON structure like
	//   {"$ion_annotation$x":y}
	// where x is the annotation label and
	// y is the annotation value.
	// Annotation structures will always have
	// exactly one field, and that fields name
	// will always begin with "$ion_annotation$"
	// followed by the annotation label.
	ShowAnnotations bool
	// contains filtered or unexported fields
}

JSONWriter is an io.WriteCloser that performs inline translation of chunks of ion data into JSON objects. See NewJSONWriter.

func NewJSONWriter 

func NewJSONWriter(w io.Writer, sep byte) *JSONWriter

NewJSONWriter creates a new JSON writer which writes either NDJSON or a JSON array depending on the value of sep: 

If sep is '\n', then the returned JSONWriter
writes NDJSON lines from each input object,
and the Close method is a no-op.

If sep is ',', then the return JSONWriter
writes a JSON array containing all the ion
values passed to Write. The call to Close
writes the final ']' byte.

NewJSONWriter will panic if sep is not one of the recognized bytes.

func (*JSONWriter) Close 

func (w *JSONWriter) Close() error

func (*JSONWriter) Write 

func (w *JSONWriter) Write(src []byte) (int, error)

Write implements io.Writer

The buffer passed to Write must contain complete ion objects.

type List 

type List struct {
	// contains filtered or unexported fields
}

List is an ion list datum

func NewList 

func NewList(st *Symtab, items []Datum) List

func (List) Datum 

func (l List) Datum() Datum

func (List) Each 

func (l List) Each(fn func(Datum) error) error

Each iterates over each datum in the list and calls fn on each datum in order. Each returns when it encounters an internal error (due to malformed ion) or when fn returns a non-nil error.

func (List) Encode 

func (l List) Encode(dst *Buffer, st *Symtab)

func (List) Equal 

func (l List) Equal(l2 List) bool

func (List) IsEmpty 

func (l List) IsEmpty() bool

func (List) Items 

func (l List) Items(items []Datum) []Datum

func (List) Iterator 

func (l List) Iterator() (Iterator, error)

Iterator returns an iterator which can be used to iterator over the items in the list.

func (List) Len 

func (l List) Len() int

Len returns the number of items in the list. If the list is malformed, the results are undefined.

type Ranges 

type Ranges struct {
	// contains filtered or unexported fields
}

func (*Ranges) AddTime 

func (rs *Ranges) AddTime(p Symbuf, t date.Time)

AddTime adds a time value to the range tracker.

func (*Ranges) AddTruncatedTime 

func (rs *Ranges) AddTruncatedTime(p Symbuf, t date.Time, trunc TimeTrunc)

AddTruncatedTime adds a truncated time value to the range tracker.

type Struct 

type Struct struct {
	// contains filtered or unexported fields
}

Struct is an ion structure datum

func NewStruct 

func NewStruct(st *Symtab, f []Field) Struct

func (Struct) Datum 

func (s Struct) Datum() Datum

func (Struct) Each 

func (s Struct) Each(fn func(Field) error) error

Each calls fn for each field in the struct. If fn returns Stop, Each stops and returns nil. If fn returns any other non-nil error, Each stops and returns that error. If Each encounters a malformed field while unpacking the struct, Each returns a non-nil error.

func (Struct) Encode 

func (s Struct) Encode(dst *Buffer, st *Symtab)

func (Struct) Equal 

func (s Struct) Equal(s2 Struct) bool

func (Struct) Field 

func (s Struct) Field(x Symbol) (Field, bool)

func (Struct) FieldByName 

func (s Struct) FieldByName(name string) (Field, bool)

func (Struct) Fields 

func (s Struct) Fields(fields []Field) []Field

Fields appends fields to the given slice and returns the appended slice.

func (*Struct) IsEmpty 

func (s *Struct) IsEmpty() bool

func (Struct) Len 

func (s Struct) Len() int

func (Struct) WithField 

func (s Struct) WithField(f Field) Struct

WithField adds or overwrites a field in s and returns a new Struct with the updated field. If a field with f.Label is already present in the structure, it is overwritten with f. Otherwise, f is added to the existing fields.

type Symbol 

type Symbol uint

Symbol represents an ion Symbol 

const (
	// SystemSymImports is the pre-interned symbol for "imports"
	SystemSymImports Symbol = 6
	// SystemSymSymbols is the pre-interned symbol for "symbols"
	SystemSymSymbols Symbol = 7
	// SystemSymSymbolTable is the pre-interned symbol for "$ion_symbol_table"
	SystemSymSymbolTable Symbol = 3
)

func ReadAnnotation 

func ReadAnnotation(buf []byte) (Symbol, []byte, []byte, error)

ReadAnnotation reads an annotation and returns the associated label, the contents of the annotation, and the remaining bytes in buf (in that order).

func ReadLabel 

func ReadLabel(msg []byte) (Symbol, []byte, error)

ReadLabel reads a symbol preceding a structure field and returns the subsequent message bytes.

func ReadSymbol 

func ReadSymbol(msg []byte) (Symbol, []byte, error)

ReadSymbol reads an ion symbol from msg and returns the subsequent message bytes, or an error if one is encountered.

func (Symbol) Encode 

func (u Symbol) Encode(dst *Buffer, st *Symtab)

func (Symbol) Type 

func (u Symbol) Type() Type

type Symbuf 

type Symbuf []byte

Symbuf is an encoded list of symtab indices.

func (*Symbuf) Prepare 

func (b *Symbuf) Prepare(n int)

Prepare the buffer to have n symbols pushed. This also clears the buffer.

func (*Symbuf) Push 

func (b *Symbuf) Push(sym Symbol)

Push adds a new symbol to the buffer. Prepare should be called first to ensure the capacity of the buffer is sufficient to accept all pushed symbols, or this method will panic.

type Symtab 

type Symtab struct {
	// contains filtered or unexported fields
}

Symtab is an ion symbol table

func (*Symtab) CloneInto 

func (s *Symtab) CloneInto(o *Symtab)

CloneInto performs a deep copy of s into o. CloneInto takes care to use some of the existing storage in o in order to reduce the copying overhead.

func (*Symtab) Contains 

func (s *Symtab) Contains(inner *Symtab) bool

Contains returns true if s is a superset of the symbols within inner, and all of the symbols in inner have the same symbol ID in s.

If x.Contains(y), then x is a semantically equivalent substitute for y.

func (*Symtab) Equal 

func (s *Symtab) Equal(o *Symtab) bool

Equal checks if two symtabs are equal.

func (*Symtab) Get 

func (s *Symtab) Get(x Symbol) string

Get gets the string associated with the given interned symbol, or returns the empty string when there is no symbol with the given association.

func (*Symtab) Intern 

func (s *Symtab) Intern(x string) Symbol

Intern interns the given string if it is not already interned and returns the associated Symbol

func (*Symtab) InternBytes 

func (s *Symtab) InternBytes(buf []byte) Symbol

InternBytes is identical to Intern, except that it accepts a []byte instead of a string as an argument.

func (*Symtab) Lookup 

func (s *Symtab) Lookup(x Symbol) (string, bool)

Lookup gets the string associated with the given interned symbol. This returns ("", false) when the symbol is not present in the table.

func (*Symtab) Marshal 

func (s *Symtab) Marshal(dst *Buffer, withBVM bool)

Marshal marshals the Symtab into 'dst' optionally with a BVM prefix.

If withBVM is false and the symbol table is empty, then no data is written to dst.

func (*Symtab) MarshalPart 

func (s *Symtab) MarshalPart(dst *Buffer, starting Symbol)

MarshalPart writes a symbol table to dst with all the symbols starting at starting. If there are no symbols above starting, then MarshalPart does not write any data.

Callers can use a previous result of s.MaxID plus MarshalPart to write incremental changes to symbol tables to an ion stream.

func (*Symtab) MaxID 

func (s *Symtab) MaxID() int

MaxID returns the total number of interned symbols. Note that ion defines ten symbols that are automatically interned, so an "empty" symbol table has MaxID() of 10.

func (*Symtab) Merge 

func (s *Symtab) Merge(o *Symtab) (modified bool, ok bool)

Merge adds new symbols from symtab `o` providing that the common symbols of the both symtabs are the same.

Returns whether merge was OK. If it was, return if new symbols were added.

func (*Symtab) Reset 

func (s *Symtab) Reset()

Reset resets a symbol table so that it no longer contains any symbols (except for the ion pre-defined symbols).

func (*Symtab) String 

func (s *Symtab) String() string

func (*Symtab) Symbolize 

func (s *Symtab) Symbolize(x string) (Symbol, bool)

Symbolize returns the symbol associated with the string 'x' in the symbol table, or (0, false) if the string has not been interned.

func (*Symtab) SymbolizeBytes 

func (s *Symtab) SymbolizeBytes(x []byte) (Symbol, bool)

SymbolizeBytes works identically to Symbolize, except that it accepts a []byte.

func (*Symtab) Truncate 

func (s *Symtab) Truncate(max int)

Truncate truncates the symbol table to the number of symbols indicated by max. Truncate can be used to restore a Symtab to its previous state as indicated by s.MaxID(). Truncate panics if max is below the number of pre-interned "system" symbols (10).

func (*Symtab) Unmarshal 

func (s *Symtab) Unmarshal(src []byte) ([]byte, error)

Unmarshal unmarshals a symbol table from 'src' into 's'. If 'src' begins with a BVM (see IsBVM), then any contents of the symbol table will be cleared before interning the new symbol values. Otherwise, the new symbols will be interned with IDs above the presently-interned symbols.

BUGS: Support for ion "shared" symbol tables is not yet implemented.

type TimeTrunc 

type TimeTrunc int
const (
	TruncToYear TimeTrunc = iota
	TruncToMonth
	TruncToDay
	TruncToHour
	TruncToMinute
	TruncToSecond
)

type Type 

type Type byte

Type is one of the ion datatypes 

const (
	NullType Type = iota
	BoolType
	UintType // unsigned integer
	IntType  // signed integer; always negative
	FloatType
	DecimalType
	TimestampType
	SymbolType
	StringType
	ClobType
	BlobType
	ListType
	SexpType
	StructType
	AnnotationType
	ReservedType
	InvalidType = Type(0xff)
)

func DecodeTLV 

func DecodeTLV(b byte) (t Type, l byte)

DecodeTLV explodes TLV byte into: type (t), raw length (l)

func Peek 

func Peek(r *bufio.Reader) (Type, int, error)

Peek peeks at the type and size of the next object in 'r'.

The returned size can be used to read only the next object from the buffer. For example: 

t, s, _ := Peek(r)
buf := make([]byte, s)
io.ReadFull(r, buf)

func TypeOf 

func TypeOf(msg []byte) Type

TypeOf returns the type of the next object in the buffer

func (Type) Composite 

func (t Type) Composite() bool

Composite returns whether or not the type is an object containing other objects.

func (Type) Integer 

func (t Type) Integer() bool

Integer returns whether or not the type is an integer type (either IntType or UintType).

func (Type) String 

func (t Type) String() string

type TypeError 

type TypeError struct {
	Wanted, Found Type
	Func, Field   string
}

TypeError is the error returned by functions when the concrete type of a datum does not match the type expected by the function.

func (*TypeError) Error 

func (t *TypeError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
Package blockfmt implements routines for reading and writing compressed and aligned ion blocks to/from backing storage.
 Package blockfmt implements routines for reading and writing compressed and aligned ion blocks to/from backing storage.
Package versify implements an ion "versifier:" code that performs procedural data generation based on example input.
 Package versify implements an ion "versifier:" code that performs procedural data generation based on example input.
Package zion implements a "zipped" ion encoding that compresses streams of ion structures in a manner such that fields within structures in the stream can be decompressed without decompressing the entire input stream.
 Package zion implements a "zipped" ion encoding that compresses streams of ion structures in a manner such that fields within structures in the stream can be decompressed without decompressing the entire input stream.
iguana
Package iguana implements a Lizard-derived compression/decompression pipeline
 Package iguana implements a Lizard-derived compression/decompression pipeline
zll
Package zll exposes types and procedures related to low-level zion decoding.
 Package zll exposes types and procedures related to low-level zion decoding.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.