ztlv

package module
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2026 License: MIT Imports: 7 Imported by: 0

README

ztlv

Go Reference Go Report Card

ztlv is a secure, low-allocation Tag-Length-Value (TLV) parser for Go.

It is designed for network protocols, critical infrastructure, and high-throughput systems where memory pressure and input validation are paramount. It operates directly on io.Reader and io.Writer interfaces, enabling streaming of datasets larger than available RAM.

Design Principles

  • Security First: Strict bounds checking prevents OOM (Out-Of-Memory) attacks. Integers are checked for overflows.
  • Low Allocation: Primitives (Tags, Lengths, Time) are encoded/decoded using stack-allocated scratchpads.
  • Optimized Strings: Uses unsafe.String (Go 1.20+) to convert byte buffers to strings without additional heap copying.
  • Robust Time: Timestamps are stored as 96-bit (64-bit Seconds + 32-bit Nanoseconds) to prevent Y2262 issues and maintain precision.
  • Big Endian: All numeric values conform to standard network byte order.

Installation

go get github.com/eraceo/ztlv

Usage

Encoder

The encoder wraps an io.Writer. It performs no heap allocations for primitive types.

package main

import (
	"bytes"
	"time"
	"github.com/eraceo/ztlv"
)

func main() {
	var buf bytes.Buffer
	enc := ztlv.NewEncoder(&buf)

	// Write Tag (0x01) + Length + String
	_ = enc.WriteTLVString(0x01, "admin")

	// Write Tag (0x02) + Length + Time (12 bytes)
	_ = enc.WriteTLVTime(0x02, time.Now())
}
Decoder

The decoder wraps an io.Reader and enforces configurable limits.

package main

import (
	"bytes"
	"log"
	"github.com/eraceo/ztlv"
)

func main() {
	// r is your io.Reader (e.g., net.Conn or bytes.Buffer)
	dec := ztlv.NewDecoder(r)

	// Decode loop
	tag, err := dec.ReadTag()
	if err != nil {
		log.Fatal(err)
	}

	switch tag {
	case 0x01:
		// Efficiently reads string (1 allocation only)
		val, _ := dec.ReadString()
		log.Printf("User: %s", val)
	case 0x02:
		// Efficiently reads Length + Time (0 allocation)
		// Uses ReadTimePrefixed because WriteTLVTime includes a length.
		ts, _ := dec.ReadTimePrefixed()
		log.Printf("Timestamp: %s", ts)
	default:
		// SkipTLV reads Length + discards payload — no allocation
		_ = dec.SkipTLV()
	}
}
Advanced: Zero-Allocation Byte Reading

For hot paths where garbage collection pressure must be eliminated, use ReadBytesInto with a reused buffer. This method separates length reading from payload reading to ensure stream integrity.

// Pre-allocated buffer (or retrieved from sync.Pool)
buf := make([]byte, 4096)
// 1. Read the length first
length, err := dec.ReadLength()
if err != nil {
    handleError(err)
}

// 2. Check if buffer is large enough
if uint32(cap(buf)) < length {
    // Handle error, or grow buffer
    handleError(fmt.Errorf("buffer too small"))
}

// 3. Read payload directly into the slice (Zero Allocation)
n, err := dec.ReadBytesInto(length, buf)
if err != nil {
    handleError(err)
}
process(buf[:n])

Security Configuration

To prevent Denial of Service (DoS) attacks via malicious length prefixes, the decoder enforces default sanity limits. These can be adjusted per instance.

dec := ztlv.NewDecoder(r)

// Adjust limits based on your protocol requirements
dec.MaxStringSize = 64 * 1024       // Limit strings to 64KB
dec.MaxBytesSize  = 10 * 1024 * 1024 // Limit raw blobs to 10MB
dec.MaxListCount  = 5000            // Limit list items

Binary Format Specification

Component Size Type Description
Tag 1 byte uint8 Identifier (0-255).
Length 4 bytes uint32 Payload size in bytes (Big Endian).
Value N bytes Raw Payload data.

Time Format: Stored as 12 bytes.

  • Bytes 0-7: Seconds (int64, Big Endian)
  • Bytes 8-11: Nanoseconds (uint32, Big Endian)

Performance

Benchmarks on AMD Ryzen 9 7900 (amd64, Go 1.22):

Operation Time Allocs
WriteUint64 4.2 ns 0
ReadUint64 6.6 ns 0
WriteString (~38B) 12.5 ns 0
ReadString (~38B) 25.7 ns 1
ReadBytesInto (1KB, reuse) 21.4 ns 0
SkipTLV (256B) 16.8 ns 0
Roundtrip uint64 32.5 ns 0
Roundtrip message (4 fields) 139 ns 1
  • Primitives (uint8 → uint64, bool, time): 0 allocs/op via stack-allocated scratch buffer.
  • Strings: 1 alloc/op — buffer only, no extra copy thanks to unsafe.String.
  • Bytes: 1 alloc/op, or 0 with ReadBytesInto + sync.Pool.
  • Skip: 0 allocs/op — uses io.Seeker fast-path on in-memory streams, pre-allocated LimitedReader fallback on network streams.
  • Encoder/Decoder: reusable via Reset() — eliminates allocations in hot loops.

Documentation

Index

Constants

View Source 
const (
	// Default safety limits
	DefaultMaxStringSize = 1 << 20  // 1 MB
	DefaultMaxBytesSize  = 10 << 20 // 10 MB
	DefaultMaxListCount  = 1000
	DefaultMaxSkipSize   = 10 << 20 // 10 MB
)

Variables

View Source 
var (
	ErrPayloadTooLarge = errors.New("ztlv: payload exceeds configured limit")
	ErrListTooLarge    = errors.New("ztlv: list count exceeds configured limit")
	ErrShortBuffer     = errors.New("ztlv: buffer too short")
	ErrInvalidTag      = errors.New("ztlv: invalid tag found")
	ErrInvalidLength   = errors.New("ztlv: invalid length for type")
)

Functions

This section is empty.

Types

type Decoder 

type Decoder struct {
	MaxStringSize uint32
	MaxBytesSize  uint32
	MaxListCount  uint32
	// This prevents blocking on malicious huge unknown tags.
	MaxSkipSize uint32
	// contains filtered or unexported fields
}

func NewDecoder 

func NewDecoder(r io.Reader) *Decoder

func (*Decoder) ReadBool 

func (d *Decoder) ReadBool() (bool, error)

ReadBool reads a boolean (1 byte).

func (*Decoder) ReadBytes 

func (d *Decoder) ReadBytes() ([]byte, error)

ReadBytes is optimized for a single allocation. It is "Low Alloc" but safe (returns a clean copy).

func (*Decoder) ReadBytesInto 

func (d *Decoder) ReadBytesInto(length uint32, buf []byte) (int, error)

ReadBytesInto enables "Zero Alloc" scenarios if the caller reuses their buffer. Warning: This method reads strictly 'length' bytes. It does NOT read the length prefix itself. The caller must have read the length beforehand (e.g. via ReadLength()). It returns the number of bytes read (which is always equal to length on success) and any error.

func (*Decoder) ReadFloat64 

func (d *Decoder) ReadFloat64() (float64, error)

ReadFloat64 reads a float64 (8 bytes, IEEE 754, Big Endian).

func (*Decoder) ReadInt64 

func (d *Decoder) ReadInt64() (int64, error)

ReadInt64 reads an int64 (8 bytes, Big Endian).

func (*Decoder) ReadLength 

func (d *Decoder) ReadLength() (uint32, error)

func (*Decoder) ReadNested 

func (d *Decoder) ReadNested(expected Tag, fn func(*Decoder) error) error

ReadNested handles a TLV container (a TLV that contains other TLVs). It reads the Tag and Length, creates a limited Decoder for the body, and executes the callback.

The nested Decoder is reused from d.nestedDec to avoid a heap allocation per call. Any unread bytes within the container are drained automatically after the callback returns, ensuring the parent stream remains correctly positioned.

func (*Decoder) ReadString 

func (d *Decoder) ReadString() (string, error)

ReadString uses "unsafe" optimizations (Go 1.20+) to reduce allocations.

func (*Decoder) ReadStrings 

func (d *Decoder) ReadStrings() ([]string, error)

ReadStrings decodes a list of strings.

NOTE: Returns nil (not []string{}) for an empty list — this is intentional and idiomatic in Go. Callers should use len(strs) == 0 rather than strs == nil to test for emptiness.

This differs from ReadBytes which returns []byte{} for empty payloads, because a nil byte slice can cause unexpected downstream panics; a nil string slice cannot.

This asymmetry is a known API rough edge. Always use len(result) == 0 for both types to write forward-compatible code.

func (*Decoder) ReadTLVBool 

func (d *Decoder) ReadTLVBool(expected Tag) (bool, error)

ReadTLVBool reads a Tag, verifies it, reads Length (must be 1), then reads Bool.

func (*Decoder) ReadTLVBytes 

func (d *Decoder) ReadTLVBytes(expected Tag) ([]byte, error)

ReadTLVBytes reads a Tag, verifies it, reads Length, then reads Bytes.

func (*Decoder) ReadTLVBytesInto added in v1.1.0 

func (d *Decoder) ReadTLVBytesInto(expected Tag, buf []byte) (int, error)

ReadTLVBytesInto reads a Tag, verifies it, reads Length, checks buffer size, and reads bytes directly into the provided buffer. This is the most efficient and secure way to read bytes without allocation.

func (*Decoder) ReadTLVFloat64 

func (d *Decoder) ReadTLVFloat64(expected Tag) (float64, error)

ReadTLVFloat64 reads a Tag, verifies it, reads Length (must be 8), then reads Float64.

func (*Decoder) ReadTLVInt64 

func (d *Decoder) ReadTLVInt64(expected Tag) (int64, error)

ReadTLVInt64 reads a Tag, verifies it, reads Length (must be 8), then reads Int64.

func (*Decoder) ReadTLVString 

func (d *Decoder) ReadTLVString(expected Tag) (string, error)

ReadTLVString reads a Tag, verifies it, reads Length, then reads String.

func (*Decoder) ReadTLVStrings added in v1.1.2 

func (d *Decoder) ReadTLVStrings(expected Tag) ([]string, error)

ReadTLVStrings reads a Tag, verifies it, then reads the list of strings.

func (*Decoder) ReadTLVTime 

func (d *Decoder) ReadTLVTime(expected Tag) (time.Time, error)

ReadTLVTime reads a Tag, verifies it, reads Length (must be 12), then reads Time.

func (*Decoder) ReadTLVUint8 

func (d *Decoder) ReadTLVUint8(expected Tag) (uint8, error)

ReadTLVUint8 reads a Tag, verifies it, reads Length (must be 1), then reads Uint8.

func (*Decoder) ReadTLVUint16 

func (d *Decoder) ReadTLVUint16(expected Tag) (uint16, error)

ReadTLVUint16 reads a Tag, verifies it, reads Length (must be 2), then reads Uint16.

func (*Decoder) ReadTLVUint32 

func (d *Decoder) ReadTLVUint32(expected Tag) (uint32, error)

ReadTLVUint32 reads a Tag, verifies it, reads Length (must be 4), then reads Uint32.

func (*Decoder) ReadTLVUint64 

func (d *Decoder) ReadTLVUint64(expected Tag) (uint64, error)

ReadTLVUint64 reads a Tag, verifies it, reads Length (must be 8), then reads Uint64.

func (*Decoder) ReadTag 

func (d *Decoder) ReadTag() (Tag, error)

func (*Decoder) ReadTime 

func (d *Decoder) ReadTime() (time.Time, error)

ReadTime decodes a robust timestamp (Seconds + Nanoseconds). Note: This reads strictly 12 bytes. It does NOT read a length prefix. Use ReadTimePrefixed if you are using standard TLV (WriteTLVTime).

func (*Decoder) ReadTimePrefixed 

func (d *Decoder) ReadTimePrefixed() (time.Time, error)

ReadTimePrefixed reads a Length prefix, verifies it is 12, and then reads the Time. This is the secure counterpart to WriteTLVTime.

func (*Decoder) ReadUint8 

func (d *Decoder) ReadUint8() (uint8, error)

ReadUint8 reads a uint8 (1 byte).

func (*Decoder) ReadUint16 

func (d *Decoder) ReadUint16() (uint16, error)

ReadUint16 reads a uint16 (2 bytes, Big Endian).

func (*Decoder) ReadUint32 

func (d *Decoder) ReadUint32() (uint32, error)

ReadUint32 reads a uint32 (4 bytes, Big Endian).

func (*Decoder) ReadUint64 

func (d *Decoder) ReadUint64() (uint64, error)

ReadUint64 reads a uint64 (8 bytes, Big Endian).

func (*Decoder) Reset added in v1.2.0 

func (d *Decoder) Reset(r io.Reader)

func (*Decoder) Skip 

func (d *Decoder) Skip(n uint32) error

Skip discards n bytes from the stream.

func (*Decoder) SkipTLV added in v1.2.0 

func (d *Decoder) SkipTLV() error

SkipTLV is a convenience helper to avoid error-prone manual ReadLength+Skip patterns in decode loops when encountering unknown tags.

func (*Decoder) VerifyTag 

func (d *Decoder) VerifyTag(expected Tag) error

VerifyTag reads a Tag and returns an error if it does not match the expected Tag.

type Encoder 

type Encoder struct {
	// contains filtered or unexported fields
}

Encoder remains lightweight. Allocations are primarily handled by the underlying Writer.

func NewEncoder 

func NewEncoder(w io.Writer) *Encoder

func (*Encoder) Reset added in v1.2.0 

func (e *Encoder) Reset(w io.Writer)

func (*Encoder) WriteBool 

func (e *Encoder) WriteBool(v bool) error

WriteBool writes a boolean (1 byte). 0x01 for true, 0x00 for false.

func (*Encoder) WriteBytes 

func (e *Encoder) WriteBytes(b []byte) error

WriteBytes writes a standard byte slice.

func (*Encoder) WriteFloat64 

func (e *Encoder) WriteFloat64(v float64) error

WriteFloat64 writes a float64 (8 bytes, IEEE 754, Big Endian).

func (*Encoder) WriteInt64 

func (e *Encoder) WriteInt64(v int64) error

WriteInt64 writes an int64 (8 bytes, Big Endian).

func (*Encoder) WriteLength 

func (e *Encoder) WriteLength(length uint32) error

func (*Encoder) WriteString 

func (e *Encoder) WriteString(s string) error

func (*Encoder) WriteStrings 

func (e *Encoder) WriteStrings(strs []string) error

func (*Encoder) WriteTLVBool 

func (e *Encoder) WriteTLVBool(tag Tag, v bool) error

WriteTLVBool writes a Tag, Length (1), then the Bool (1 byte).

func (*Encoder) WriteTLVBytes 

func (e *Encoder) WriteTLVBytes(tag Tag, b []byte) error

WriteTLVBytes writes a Tag, then the Length of the bytes, then the Bytes themselves.

func (*Encoder) WriteTLVFloat64 

func (e *Encoder) WriteTLVFloat64(tag Tag, v float64) error

WriteTLVFloat64 writes a Tag, Length (8), then the Float64 (8 bytes).

func (*Encoder) WriteTLVInt64 

func (e *Encoder) WriteTLVInt64(tag Tag, v int64) error

WriteTLVInt64 writes a Tag, Length (8), then the Int64 (8 bytes).

func (*Encoder) WriteTLVString 

func (e *Encoder) WriteTLVString(tag Tag, s string) error

WriteTLVString writes a Tag, then the Length of the string, then the String itself.

func (*Encoder) WriteTLVStrings added in v1.1.2 

func (e *Encoder) WriteTLVStrings(tag Tag, strs []string) error

WriteTLVStrings writes a Tag, then the list of strings (count + each string TLV).

func (*Encoder) WriteTLVTime 

func (e *Encoder) WriteTLVTime(tag Tag, t time.Time) error

WriteTLVTime writes a Tag, then the Length of the time (12 bytes), then the Time itself.

func (*Encoder) WriteTLVUint8 

func (e *Encoder) WriteTLVUint8(tag Tag, v uint8) error

WriteTLVUint8 writes a Tag, Length (1), then the Uint8 (1 byte).

func (*Encoder) WriteTLVUint16 

func (e *Encoder) WriteTLVUint16(tag Tag, v uint16) error

WriteTLVUint16 writes a Tag, Length (2), then the Uint16 (2 bytes).

func (*Encoder) WriteTLVUint32 

func (e *Encoder) WriteTLVUint32(tag Tag, v uint32) error

WriteTLVUint32 writes a Tag, Length (4), then the Uint32 (4 bytes).

func (*Encoder) WriteTLVUint64 

func (e *Encoder) WriteTLVUint64(tag Tag, v uint64) error

WriteTLVUint64 writes a Tag, Length (8), then the Uint64 (8 bytes).

func (*Encoder) WriteTag 

func (e *Encoder) WriteTag(tag Tag) error

func (*Encoder) WriteTime 

func (e *Encoder) WriteTime(t time.Time) error

WriteTime encodes a timestamp robustly: compatible with all dates, avoids Year 2038 issues.

func (*Encoder) WriteUint8 

func (e *Encoder) WriteUint8(v uint8) error

WriteUint8 writes a uint8 (1 byte).

func (*Encoder) WriteUint16 

func (e *Encoder) WriteUint16(v uint16) error

WriteUint16 writes a uint16 (2 bytes, Big Endian).

func (*Encoder) WriteUint32 

func (e *Encoder) WriteUint32(v uint32) error

WriteUint32 writes a uint32 (4 bytes, Big Endian).

func (*Encoder) WriteUint64 

func (e *Encoder) WriteUint64(v uint64) error

WriteUint64 writes a uint64 (8 bytes, Big Endian).

type Tag 

type Tag byte

Tag is a single-byte TLV identifier (0x00–0xFF). NOTE: 256 possible tags is a known limitation of this format. If your protocol requires more identifiers, consider a two-byte tag or varint encoding — this is a planned future extension.

type UnexpectedTagError 

type UnexpectedTagError struct {
	Expected Tag
	Actual   Tag
}

UnexpectedTagError is returned when the read tag does not match the expected tag.

func (*UnexpectedTagError) Error 

func (e *UnexpectedTagError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.