codec

package
v0.0.0-...-4692381 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 16, 2026 License: MIT Imports: 17 Imported by: 2

Documentation

Overview

Package codec provides a unified interface for encoding/decoding HTTP payloads with support for both JSON and Cowrie formats via content-type negotiation.

Index

Constants

View Source 
const (
	ContentTypeJSON   = "application/json"
	ContentTypeCowrie = "application/cowrie"
)

Content-Type constants for negotiation

View Source 
const (
	// MasterMagic is the 4-byte magic header for Cowrie stream frames.
	// Distinct from Cowrie document magic ('SJ') and legacy stream ([u32 len]).
	MasterMagic0 = 'S'
	MasterMagic1 = 'J'
	MasterMagic2 = 'S'
	MasterMagic3 = 'T'

	// MasterVersion is the current master stream version.
	MasterVersion = 0x02
)

Master Stream wire format constants

View Source 
const (
	// FlagMasterCompressed indicates the payload is compressed.
	FlagMasterCompressed = 0x01

	// FlagMasterCRC indicates a CRC32 checksum follows the payload.
	FlagMasterCRC = 0x02

	// FlagMasterDeterministic indicates the payload was encoded deterministically.
	FlagMasterDeterministic = 0x04

	// FlagMasterMeta indicates metadata is present before the payload.
	FlagMasterMeta = 0x08

	// Compression type bits (bits 4-5)
	FlagMasterCompGzip = 0x10 // compression = gzip
	FlagMasterCompZstd = 0x20 // compression = zstd
)

Master stream frame flags

View Source 
const MaxDecompressedSize = 256 * 1024 * 1024

MaxDecompressedSize is the maximum allowed decompressed payload size (256 MB). This prevents decompression bombs where a small compressed payload expands into gigabytes of RAM.

Variables

View Source 
var (
	ErrMasterInvalidMagic   = errors.New("cowrie master: invalid magic")
	ErrMasterInvalidVersion = errors.New("cowrie master: unsupported version")
	ErrMasterCRCMismatch    = errors.New("cowrie master: CRC mismatch")
	ErrMasterTruncated      = errors.New("cowrie master: truncated frame")
)

Master stream errors

Functions

func DecodeBytes 

func DecodeBytes(data []byte, v any) error

DecodeBytes is a convenience function for decoding Cowrie bytes directly.

func DecodeFloat32Tensor 

func DecodeFloat32Tensor(v *cowrie.Value) []float32

DecodeFloat32Tensor converts an Cowrie Tensor value back to a float32 slice. Returns nil if the value is not a tensor or has wrong dtype.

func DecodeFloat64Tensor 

func DecodeFloat64Tensor(v *cowrie.Value) []float64

DecodeFloat64Tensor converts an Cowrie Tensor value back to a float64 slice. Returns nil if the value is not a tensor or has wrong dtype.

func DecodeStringArray 

func DecodeStringArray(v *cowrie.Value) []string

DecodeStringArray decodes an Cowrie array to []string.

func DecodeStringMapArray 

func DecodeStringMapArray(v *cowrie.Value) []map[string]string

DecodeStringMapArray decodes an Cowrie array of objects to []map[string]string.

func DecodeTensorAuto 

func DecodeTensorAuto(v *cowrie.Value) any

DecodeTensorAuto decodes a tensor value respecting its dtype. Returns the appropriate Go slice type based on the tensor's dtype. For float32 tensors returns []float32, for float64 returns []float64, etc.

func EncodeBytes 

func EncodeBytes(v any) ([]byte, error)

EncodeBytes is a convenience function for encoding to Cowrie bytes directly.

func EncodeFloat32Tensor 

func EncodeFloat32Tensor(values []float32) *cowrie.Value

EncodeFloat32Tensor converts a float32 slice to an Cowrie Tensor value. This is ~48% smaller than JSON's text representation of floats. JSON: "scores": [0.123456, 0.789012, ...] = 10-20 bytes per float Cowrie Tensor: 4 bytes per float (native binary)

func EncodeFloat64AsFloat32Tensor 

func EncodeFloat64AsFloat32Tensor(values []float64) *cowrie.Value

EncodeFloat64AsFloat32Tensor converts float64 slice to float32 tensor for 60% savings. Use when precision loss is acceptable (e.g., ML features, embeddings).

func EncodeFloat64Tensor 

func EncodeFloat64Tensor(values []float64) *cowrie.Value

EncodeFloat64Tensor converts a float64 slice to an Cowrie Tensor value. This is ~55% smaller than JSON's text representation of floats. JSON: "scores": [0.123456789012, ...] = 15-25 bytes per float Cowrie Tensor: 8 bytes per float (native binary float64)

func FastEncode 

func FastEncode(v any) ([]byte, error)

FastEncode encodes v to Cowrie using cached struct encoders. This is 2-3x faster than the generic path for struct types.

func FastEncodeBytes 

func FastEncodeBytes(v any) ([]byte, error)

FastEncodeBytes is the pooled-buffer version of FastEncode.

func GetFloat32Array 

func GetFloat32Array(v *cowrie.Value, key string) []float32

GetFloat32Array safely gets a float32 array from an object by key. Handles both tensor encoding and regular arrays.

func GetString 

func GetString(v *cowrie.Value, key string) string

GetString safely gets a string value from an object by key.

func GetStringArray 

func GetStringArray(v *cowrie.Value, key string) []string

GetStringArray safely gets a string array from an object by key.

func GetStructInfo 

func GetStructInfo(t reflect.Type) *structInfo

GetStructInfo is the public accessor for struct type information.

func IsCowrieDocument 

func IsCowrieDocument(data []byte) bool

IsCowrieDocument checks if data starts with Cowrie document magic. Returns false for master stream format (which also starts with 'SJ' but has 'ST' after).

func IsFloat32Array 

func IsFloat32Array(v any) ([]float32, bool)

IsFloat32Array checks if a Go value is a []float32 that would benefit from tensor encoding.

func IsLegacyStream 

func IsLegacyStream(data []byte) bool

IsLegacyStream checks if data starts with legacy stream format.

func IsMasterStream 

func IsMasterStream(data []byte) bool

IsMasterStream checks if data starts with master stream magic.

func IsRegistered 

func IsRegistered[T any]() bool

IsRegistered returns true if a fast unmarshaler is registered for the type.

func ReadAllStream 

func ReadAllStream[T any](data []byte) ([]T, error)

ReadAll reads all records from the stream into a slice.

func Register 

func Register[T any](unmarshaler func(*cowrie.Value, *T) error)

Register adds a fast unmarshaler for a specific type. The unmarshaler function receives the Cowrie value and a pointer to the target type.

Example: 

codec.Register(func(v *cowrie.Value, resp *QueryResponse) error {
    if ids := v.Get("ids"); ids != nil {
        resp.IDs = decodeStringArray(ids)
    }
    if scores := v.Get("scores"); scores != nil {
        resp.Scores = codec.DecodeFloat32Tensor(scores)
    }
    return nil
})

func TryEncodeNumericSlice 

func TryEncodeNumericSlice(arr []any) *cowrie.Value

TryEncodeNumericSlice attempts to encode []interface{} as a tensor if all elements are numeric. Returns nil if the slice is not purely numeric. Uses default precision (converts to float32).

func TryEncodeNumericSliceWithOpts 

func TryEncodeNumericSliceWithOpts(arr []any, highPrecision bool) *cowrie.Value

TryEncodeNumericSliceWithOpts attempts to encode []interface{} as a tensor with precision control. If highPrecision is true, uses float64 encoding to preserve precision.

func Unregister 

func Unregister[T any]()

Unregister removes a fast unmarshaler for a type. Useful for testing or dynamic reconfiguration.

Types

type Codec 

type Codec interface {
	// ContentType returns the MIME type for this codec (e.g., "application/json")
	ContentType() string

	// Encode encodes v and writes to w
	Encode(w io.Writer, v any) error

	// Decode reads from r and decodes into v
	Decode(r io.Reader, v any) error
}

Codec provides encoding/decoding for HTTP payloads. Implementations must be safe for concurrent use.

func FromContentType 

func FromContentType(contentType string) Codec

FromContentType selects the appropriate codec based on Content-Type header. Returns JSON codec if Content-Type is missing or not recognized (safe default).

func FromRequest 

func FromRequest(r *http.Request) Codec

FromRequest selects the appropriate codec based on the Accept header. Returns JSON codec if Accept is missing or not recognized (safe default).

type CowrieCodec 

type CowrieCodec struct {
	// HighPrecision preserves float64 precision instead of converting to float32.
	// When false (default), float64 slices are encoded as float32 for ~50% size reduction.
	// When true, float64 slices are encoded as float64 (larger but lossless).
	HighPrecision bool
}

CowrieCodec implements Codec using the Cowrie binary format. It provides native tensor encoding for float32 slices (~48% smaller).

func (CowrieCodec) ContentType 

func (CowrieCodec) ContentType() string

ContentType returns "application/cowrie".

func (CowrieCodec) Decode 

func (c CowrieCodec) Decode(r io.Reader, v any) error

Decode decodes Cowrie from r into v. Uses a tiered approach for maximum performance:

  1. Type registry (fastest) - pre-compiled unmarshalers for hot types
  2. Exact natural-type assignment - direct set when the decoded Go type matches
  3. Typed reflection unmarshal - for structs, typed maps/slices, pointers, etc.

func (CowrieCodec) Encode 

func (c CowrieCodec) Encode(w io.Writer, v any) error

Encode encodes v as Cowrie and writes to w. Float32 slices are automatically encoded as native tensors. If c.HighPrecision is true, float64 slices preserve full precision.

type FastUnmarshaler 

type FastUnmarshaler func(*cowrie.Value, any) error

FastUnmarshaler is a type-specific unmarshaler that avoids reflection overhead. Use Register[T] to add fast unmarshalers for hot types.

type JSONCodec 

type JSONCodec struct{}

JSONCodec implements Codec using standard encoding/json. This is the default codec for backward compatibility.

func (JSONCodec) ContentType 

func (JSONCodec) ContentType() string

ContentType returns "application/json".

func (JSONCodec) Decode 

func (JSONCodec) Decode(r io.Reader, v any) error

Decode decodes JSON from r into v.

func (JSONCodec) Encode 

func (JSONCodec) Encode(w io.Writer, v any) error

Encode encodes v as JSON and writes to w.

type MasterFrame 

type MasterFrame struct {
	Header  MasterFrameHeader
	Meta    *cowrie.Value // nil if no metadata
	Payload *cowrie.Value
	TypeID  uint32
}

MasterFrame represents a decoded master stream frame.

func ReadAllMasterStream 

func ReadAllMasterStream(data []byte, opts MasterReaderOptions) ([]*MasterFrame, error)

ReadAll reads all frames from the stream.

type MasterFrameHeader 

type MasterFrameHeader struct {
	Version     uint8
	Flags       uint8
	HeaderLen   uint16
	TypeID      uint32 // Schema fingerprint (low 32 bits)
	PayloadLen  uint32
	RawLen      uint32 // 0 if not compressed
	MetaLen     uint32
	Compression cowrie.Compression
}

MasterFrameHeader represents a parsed master stream frame header.

type MasterReader 

type MasterReader struct {
	// contains filtered or unexported fields
}

MasterReader reads Cowrie master stream frames.

func NewMasterReader 

func NewMasterReader(data []byte, opts MasterReaderOptions) *MasterReader

NewMasterReader creates a new master stream reader.

func (*MasterReader) Next 

func (mr *MasterReader) Next() (*MasterFrame, error)

Next reads the next frame from the stream. Returns io.EOF when no more frames.

func (*MasterReader) Position 

func (mr *MasterReader) Position() int

Position returns the current read position.

func (*MasterReader) Remaining 

func (mr *MasterReader) Remaining() int

Remaining returns the number of bytes remaining.

type MasterReaderOptions 

type MasterReaderOptions struct {
	// TypeHandlers maps type IDs to decode functions.
	TypeHandlers map[uint32]func(*cowrie.Value) (any, error)

	// MaxDecompressedSize limits decompression (prevents bombs).
	MaxDecompressedSize int

	// AllowLegacy enables reading legacy [u32 len][payload] streams.
	AllowLegacy bool
}

MasterReaderOptions configures the MasterReader.

func DefaultMasterReaderOptions 

func DefaultMasterReaderOptions() MasterReaderOptions

DefaultMasterReaderOptions returns sensible defaults.

type MasterWriter 

type MasterWriter struct {
	// contains filtered or unexported fields
}

MasterWriter writes Cowrie master stream frames.

func NewMasterWriter 

func NewMasterWriter(w io.Writer, opts MasterWriterOptions) *MasterWriter

NewMasterWriter creates a new master stream writer.

func (*MasterWriter) Write 

func (mw *MasterWriter) Write(v any) error

Write encodes a value and writes it as a master stream frame.

func (*MasterWriter) WriteWithMeta 

func (mw *MasterWriter) WriteWithMeta(v any, meta *cowrie.Value) error

WriteWithMeta encodes a value with optional metadata.

type MasterWriterOptions 

type MasterWriterOptions struct {
	// Deterministic encodes with sorted keys for reproducible output.
	Deterministic bool

	// Compression specifies the compression algorithm.
	Compression cowrie.Compression

	// EnableCRC enables CRC32 checksum for data integrity.
	EnableCRC bool

	// TypeRegistry maps type IDs to handlers for type routing.
	TypeRegistry map[uint32]string
}

MasterWriterOptions configures the MasterWriter.

func DefaultMasterWriterOptions 

func DefaultMasterWriterOptions() MasterWriterOptions

DefaultMasterWriterOptions returns sensible defaults.

type StreamReader 

type StreamReader struct {
	// contains filtered or unexported fields
}

StreamReader reads length-prefixed Cowrie records from input data.

func NewStreamReader 

func NewStreamReader(data []byte) *StreamReader

NewStreamReader creates a new Cowrie stream reader.

func (*StreamReader) Next 

func (sr *StreamReader) Next(target any) error

Next reads the next record from the stream. Returns io.EOF when no more records.

func (*StreamReader) Position 

func (sr *StreamReader) Position() int

Position returns the current read position.

func (*StreamReader) Remaining 

func (sr *StreamReader) Remaining() int

Remaining returns the number of bytes remaining to read.

type StreamWriter 

type StreamWriter struct {
	// contains filtered or unexported fields
}

StreamWriter writes length-prefixed Cowrie records to an output stream. Format: [u32 length][cowrie bytes] per record. This is the Cowrie equivalent of JSON Lines (JSONL).

func NewStreamWriter 

func NewStreamWriter(w io.Writer) *StreamWriter

NewStreamWriter creates a new Cowrie stream writer.

func (*StreamWriter) Sync 

func (sw *StreamWriter) Sync() error

Sync flushes the underlying writer if it supports Sync.

func (*StreamWriter) Write 

func (sw *StreamWriter) Write(v any) error

Write encodes a value as Cowrie and writes it as a length-prefixed frame.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.