wreck

package module
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 6, 2024 License: MIT Imports: 8 Imported by: 1

README

wreck

Wreck is a binary format for efficient streaming, storing and exchanging "vectors" that maximizes efficiency and minimizes overhead. The format annotates "vectors" with unique and sort keys making it possible to store and lookup without doing a full deserialization.

Principles

Compactness: Aim to minimize the size of the serialized data. This involves using efficient encoding schemes, and minimizing metadata overhead.

Simplicity: The format is easy to parse and generate using streaming codecs. It avoids overly complex structures and uses only byte sequences.

Efficiency: Ensure that both serialization and deserialization processes are fast. Use fixed-size data types where possible to speed up parsing, and minimize the need for complex computations or lookups.

Cross-platform Compatibility: The format uses only little endian encoding and octet-streams. It avoids any character encodings. Floats are encoded using IEEE 754 binary representation. It ensures the correctness across different platforms and architectures.

The format simplify the implementation through establishing dependencies to external stream codecs in the following aspects. It makes a limitation that codec cannot be used standalone and requires applications to negotiate these parameters.

Security: Use external stream ciphers.

Integrity: Use external streaming error detection and error correction schemas.

Compression: Use external compression. The nature of the data does not allowing extreme gains with compression. Gzip saves only 16% with best compression.

Format

//
// 0x00 : 4 byte     | Block Size    (L)
// 0x04 : 2 byte     | Vector Size   (V)
// 0x06 : 2 byte     | Sort Key Size (S)
// 0x08 : 1 byte * V | Vector
// 0xZZ : 1 byte * S | Sort Key
// 0xXX : 1 byte *   | Unique Key
//

Getting started

The latest version of the module is available at main branch. All development, including new features and bug fixes, take place on the main branch using forking and pull requests as described in contribution guidelines. The stable version is available via Golang modules.

Use go get to retrieve the library and add it as dependency to your application.

go get -u github.com/kshard/wreck
Quick Example
// Create writer for []float32 vector
w := wreck.NewWriter[float32](out)

// Writer vector
if err := w.Write(uniqueKey, sortKey, vector); err != nil {
  // ...
}

// Create scanner for []float32 vector
r := wreck.NewScanner[float32](in)

// Scan through stream
for r.Scan() {
  // consume vector  
  r.UniqueKey()
  r.Vector()
}

if err := r.Err(); err != nil {
  // ...
}
Use-cases
  • Large vector streams
    • Output Writer[T any]
    • Input Scanner[T any]
  • Batching vectors, using JSON as primary protocol
    • On-the-wire protocol encoding/decoding with WriterJSON and ReaderJSON
    • Output Writer[T any]
    • Input Scanner[T any]
  • Transmitting one vector in the packet
    • Output Encoder[T any]
    • Input Decoder[T any]

How To Contribute

The library is MIT licensed and accepts contributions via GitHub pull requests:

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

The build and testing process requires Go version 1.21 or later.

commit message

The commit message helps us to write a good release note, speed-up review process. The message should address two question what changed and why. The project follows the template defined by chapter Contributing to a Project of Git book.

bugs

If you experience any issues with the library, please let us know via GitHub issues. We appreciate detailed and accurate reports that help us to identity and replicate the issue.

License

See LICENSE

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Decode 

func Decode(r io.Reader, wreck *Chunk) error

func Encode 

func Encode(w io.Writer, wreck *Chunk) error

func NewReaderJSON added in v0.0.2 

func NewReaderJSON(r io.Reader, withGzip bool) (io.Reader, error)

Create input stream to read vectors from [json.RawMessage].

func NewWriterJSON added in v0.0.2 

func NewWriterJSON(w io.Writer, withGzip bool) io.WriteCloser

Create output stream for of vectors to be used as [json.RawMessage].

Types

type Chunk 

type Chunk struct {
	UniqueKey []uint8
	SortKey   []uint8
	Vector    []uint8
}

type Decoder added in v0.0.2 

type Decoder[T any] struct {
	// contains filtered or unexported fields
}

Decode binary packet to vector

func NewDecoder added in v0.0.2 

func NewDecoder[T any]() *Decoder[T]

func (*Decoder[T]) Decode added in v0.0.2 

func (codec *Decoder[T]) Decode(pack []byte, uniqueKey, sortKey *[]uint8, vec *[]T) (err error)

func (*Decoder) WithDecoder added in v0.0.2 

func (codec *Decoder) WithDecoder(fmap func([]uint8) []T)

func (*Decoder) WithMaxSortKey added in v0.0.2 

func (codec *Decoder) WithMaxSortKey(v int)

func (*Decoder) WithMaxUniqueKey added in v0.0.2 

func (codec *Decoder) WithMaxUniqueKey(v int)

func (*Decoder) WithMaxVector added in v0.0.2 

func (codec *Decoder) WithMaxVector(v int)

type Encoder added in v0.0.2 

type Encoder[T any] struct {
	// contains filtered or unexported fields
}

Encode single vector as binary packet

func NewEncoder added in v0.0.2 

func NewEncoder[T any]() *Encoder[T]

func (*Encoder[T]) Encode added in v0.0.2 

func (codec *Encoder[T]) Encode(uniqueKey, sortKey []uint8, vec []T) ([]byte, error)

func (*Encoder) WithEncoder added in v0.0.2 

func (codec *Encoder) WithEncoder(fmap func([]T) []uint8)

func (*Encoder) WithMaxSortKey added in v0.0.2 

func (codec *Encoder) WithMaxSortKey(v int)

func (*Encoder) WithMaxUniqueKey added in v0.0.2 

func (codec *Encoder) WithMaxUniqueKey(v int)

func (*Encoder) WithMaxVector added in v0.0.2 

func (codec *Encoder) WithMaxVector(v int)

type Scanner 

type Scanner[T any] struct {
	// contains filtered or unexported fields
}

Vector stream

func NewScanner 

func NewScanner[T any](r io.Reader) *Scanner[T]

func (*Scanner[T]) Err 

func (codec *Scanner[T]) Err() error

func (*Scanner[T]) Scan 

func (codec *Scanner[T]) Scan() bool

func (*Scanner[T]) SortKey 

func (codec *Scanner[T]) SortKey() []uint8

func (*Scanner[T]) UniqueKey 

func (codec *Scanner[T]) UniqueKey() []uint8

func (*Scanner[T]) Vector 

func (codec *Scanner[T]) Vector() []T

func (*Scanner) WithDecoder 

func (codec *Scanner) WithDecoder(fmap func([]uint8) []T)

func (*Scanner) WithMaxSortKey added in v0.0.2 

func (codec *Scanner) WithMaxSortKey(v int)

func (*Scanner) WithMaxUniqueKey added in v0.0.2 

func (codec *Scanner) WithMaxUniqueKey(v int)

func (*Scanner) WithMaxVector added in v0.0.2 

func (codec *Scanner) WithMaxVector(v int)

type Writer 

type Writer[T any] struct {
	// contains filtered or unexported fields
}

Binary stream vector writer

func NewWriter 

func NewWriter[T any](w io.Writer) *Writer[T]

Create instance of vector writer

func (*Writer) WithEncoder 

func (codec *Writer) WithEncoder(fmap func([]T) []uint8)

func (*Writer) WithMaxSortKey added in v0.0.2 

func (codec *Writer) WithMaxSortKey(v int)

func (*Writer) WithMaxUniqueKey added in v0.0.2 

func (codec *Writer) WithMaxUniqueKey(v int)

func (*Writer) WithMaxVector added in v0.0.2 

func (codec *Writer) WithMaxVector(v int)

func (*Writer[T]) Write 

func (codec *Writer[T]) Write(uniqueKey, sortKey []uint8, vec []T) error

Write vectors

type WriterJSON added in v0.0.2 

type WriterJSON []io.WriteCloser

Output stream for of vectors to be used as [json.RawMessage]. The stream is base64 encoded string "H4sI...eQCAN" compatible with Golang's []byte JSON encoding protocol.

func (WriterJSON) Close added in v0.0.2 

func (w WriterJSON) Close() error

func (WriterJSON) Write added in v0.0.2 

func (w WriterJSON) Write(p []byte) (n int, err error)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.