datacodecs

package module
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 4, 2026 License: MIT Imports: 16 Imported by: 1

README

datacodecs

Go Reference

A Go package for creating cryptographically signed data packages with configurable codec pipelines for compression, encryption, and custom transformations.

Overview

datacodecs provides a secure, versioned format for packaging arbitrary binary data for transmission across untrusted channels. Key features include:

  • Ed25519 Digital Signatures: Every package is signed to ensure integrity and authenticity
  • Configurable Codec Pipelines: Chain multiple codecs (compression, encryption, etc.) for data transformation
  • Self-Describing Format: Packages contain all metadata needed for unpacking, including the codec pipeline definition
  • Version Support: Built-in versioning for forward compatibility

Installation

go get gitlab.com/possumco/datacodecs

Usage

Creating a Package
package main

import (
  "crypto/ed25519"
  "crypto/rand"
  "fmt"
  "log"

  "gitlab.com/possumco/datacodecs"
)

func main() {
  // Generate an Ed25519 key pair
  publicKey, privateKey, err := ed25519.GenerateKey(rand.Reader)
  if err != nil {
    log.Fatal(err)
  }

  // Create a codec (compression with best compression level)
  codec, err := datacodecs.NewZlib(datacodecs.WithCompressionMode(datacodecs.BestCompression))
  if err != nil {
    log.Fatal(err)
  }

  // Original data to package
  data := []byte("Hello, World!")
  batchID := []byte("batch-001")
  segmentID := []byte("segment-001")

  // Create the package
  pkg, err := datacodecs.CreatePackage(data, codec, batchID, segmentID, privateKey, nil)
  if err != nil {
    log.Fatal(err)
  }

  // Serialize for transmission
  serialized := pkg.Bytes()
  fmt.Printf("Package (%d bytes): %s\n", len(serialized), string(serialized))

  // Store publicKey for recipient to verify and unpack
  _ = publicKey
}
Unpacking a Package
package main

import (
  "crypto/ed25519"
  "fmt"
  "log"

  "gitlab.com/possumco/datacodecs"
)

func main() {
  // Received serialized package (from transmission)
  serialized := []byte("v1.YmF0Y2gtMDAx...") // truncated for example

  // Public key of the sender (obtained through secure channel)
  var publicKey ed25519.PublicKey // = ...

  // Unpack and verify signature
  pkg, err := datacodecs.UnpackPackage(serialized, publicKey)
  if err != nil {
    log.Fatal("Failed to unpack or verify:", err)
  }

  // Reconstruct the codec from the pipeline metadata
  codec, err := pkg.CodecFromPipeline(nil)
  if err != nil {
    log.Fatal("Failed to reconstruct codec:", err)
  }

  // Decode the codec transformations to recover original data
  originalData, err := codec.Decode(pkg.Data)
  if err != nil {
    log.Fatal("Failed to recover data:", err)
  }

  fmt.Printf("Recovered data: %s\n", string(originalData))
  fmt.Printf("Batch ID: %s\n", string(pkg.Meta.BatchID))
  fmt.Printf("Segment ID: %s\n", string(pkg.Meta.SegmentID))
  fmt.Printf("Created at: %s\n", pkg.Meta.CreationUTCTimestamp)
}
Using Encryption

For packages requiring confidentiality, add an encryption codec to the pipeline:

// Create a password retriever function
retriever := func(tag string) ([]byte, error) {
  // In production, retrieve the 32-byte AES-256 key securely
  keys := map[string][]byte{
    "my-key-tag": make([]byte, 32), // Your 32-byte key here
  }
  if key, ok := keys[tag]; ok {
    return key, nil
  }
  return nil, fmt.Errorf("unknown key tag: %s", tag)
}

// Create codec chain: compress then encrypt
zlibCodec, _ := datacodecs.NewZlib()
encryptCodec, _ := datacodecs.NewEncrypterWithParent(zlibCodec, retriever, datacodecs.WithPasswordTag("my-key-tag"))

// When unpacking, pass the retriever in options
opts := map[string]any{"PasswordRetriever": retriever}
codec, err := pkg.CodecFromPipeline(opts)

Available Codecs

Codec Description
NewZlib Zlib compression with configurable compression level
NewEncrypter AES-256-GCM encryption with tag-based key retrieval
NewNoOp Pass-through codec (no transformation)
NewReverser Byte reversal (useful for testing)

Wire Format

The package serialization format is documented in the Package Format Specification, which provides an RFC-style definition including:

  • Wire format structure: <version>.<meta>.<data>.<signature>
  • Base64url encoding rules (RFC 4648)
  • Ed25519 signature generation (RFC 8032)
  • Timestamp format (RFC 3339)
  • ABNF grammar (RFC 5234)

Security

  • Packages are signed using Ed25519 for integrity and authenticity
  • Each package includes a unique 128-bit cryptographically random nonce
  • Signature verification is mandatory during unpacking
  • For confidentiality, use the encryption codec in your pipeline

License

See LICENSE for details.

Documentation

Index

Examples

Constants

View Source 
const (
	NoCompression      = zlib.NoCompression
	BestSpeed          = zlib.BestSpeed
	BestCompression    = zlib.BestCompression
	DefaultCompression = zlib.DefaultCompression
	HuffmanOnly        = zlib.HuffmanOnly
)

Variables

This section is empty.

Functions

func AddCodecParser 

func AddCodecParser(codeTypeName string, parser CodecParser) error

AddCodecParser is used to register additional Codecs

func WithCompressionMode 

func WithCompressionMode(mode int) func(o *ZlibCodecOptions)

WithCompressionMode sets the compression mode to be used

func WithPasswordTag 

func WithPasswordTag(tag string) func(*EncrypterCodecOptions)

WithPasswordTag allows specific tags to be specified for password retrieval

Types

type BatchID 

type BatchID []byte

BatchID is an identifier used by the creator of a Package to disambiguate generation, allowing successful reconstruction

type Codec 

type Codec interface {
	Pipeline() Pipeline
	Encode(b []byte) ([]byte, error)
	Decode(b []byte) ([]byte, error)
}

Codec provides the ability to step-wise transform a byte slice into a target output, and rollback that change to recover the original information

The sequence of steps is defined by the Pipeline, which will at minimum comprise this Codec instance, but it is expected this will also contain parent Codecs if these are specified within the Codec implementation.

Encode will process in sequence from the outermost parent Codec, returning after applying the transformation of this Codec, and Rollback works in the reverse order

func NewEncrypter 

func NewEncrypter(retriever PasswordRetriever, opts ...func(*EncrypterCodecOptions)) (Codec, error)

NewEncrypter returns a Codec that creates an encrypted copy of its input

Example 
data := []byte("Hello World")

retriever := func(tag string) ([]byte, error) {
	fmt.Println(tag)
	return []byte("abcdef0123456789abcdef0123456789"), nil
}

c, _ := NewEncrypter(retriever, WithPasswordTag("dummy"))

generated, err := c.Encode(data)
if err != nil {
	fmt.Println(err)
}

rollback, err := c.Decode(generated)
if err != nil {
	fmt.Println(err)
}

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
dummy
dummy
true
[{"name":"encryptCodec","params":"{\"ptag\":\"dummy\"}"}]
Example (Nonce) 
data := []byte("Hello World")

retriever := func(_ string) ([]byte, error) {
	return []byte("abcdef0123456789abcdef0123456789"), nil
}

c, _ := NewEncrypter(retriever, WithPasswordTag("dummy"))

gen1, _ := c.Encode(data)
gen2, _ := c.Encode(data)
fmt.Println(bytes.Equal(gen1, gen2))

roll1, _ := c.Decode(gen1)
roll2, _ := c.Decode(gen2)
fmt.Println(bytes.Equal(data, roll1))
fmt.Println(bytes.Equal(data, roll2))

Output:
false
true
true
Example (Pipeline) 
data := []byte("Hello World")

retriever := func(tag string) ([]byte, error) {
	return []byte("abcdef0123456789abcdef0123456789"), nil
}

reverser, _ := NewReverser()

c, _ := NewEncrypterWithParent(reverser, retriever, WithPasswordTag("dummy"))

generated, _ := c.Encode(data)

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
true
[{"name":"reverserCodec","params":""},{"name":"encryptCodec","params":"{\"ptag\":\"dummy\"}"}]

func NewEncrypterWithParent 

func NewEncrypterWithParent(parent Codec, retriever PasswordRetriever, opts ...func(*EncrypterCodecOptions)) (Codec, error)

NewEncrypterWithParent returns a Codec that has at least one precursor codec, which will return encrypted data

func NewNoOp 

func NewNoOp() (Codec, error)

NewNoOp returns a Codec that creates a copy of its input

Example 
data := []byte("Hello World")

c, _ := NewNoOp()

generated, _ := c.Encode(data)

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
true
[{"name":"noOpCodec","params":""}]

func NewNoOpWithParent 

func NewNoOpWithParent(parent Codec) (Codec, error)

NewNoOpWithParent returns a Codec that has at least one precursor codec

func NewReverser 

func NewReverser() (Codec, error)

NewReverser returns a Codec that creates a copy with its contents reversed

Example 
data := []byte("Hello World")

c, _ := NewReverser()

generated, _ := c.Encode(data)

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))

Output:
true

func NewReverserWithParent 

func NewReverserWithParent(parent Codec) (Codec, error)

NewNoOpWithParent returns a Codec that has at least one precursor codec. These will be executed in advance of this Code instance, resulting in an output that is their output reversed

Example 
data := []byte("Hello World")

parent, _ := NewReverser()

c, _ := NewReverserWithParent(parent)

generated, _ := c.Encode(data)

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
true
[{"name":"reverserCodec","params":""},{"name":"reverserCodec","params":""}]
Example (Pipeline) 
data := []byte("Hello World")

reverser, _ := NewReverser()

noOp, _ := NewNoOpWithParent(reverser)

c, _ := NewReverserWithParent(noOp)

generated, _ := c.Encode(data)

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
true
[{"name":"reverserCodec","params":""},{"name":"noOpCodec","params":""},{"name":"reverserCodec","params":""}]

func NewZlib 

func NewZlib(opts ...func(*ZlibCodecOptions)) (Codec, error)

NewZlib returns a Codec that creates a compressed copy of its input

Example 
data := []byte("Hello World")

c, _ := NewZlib()

generated, _ := c.Encode(data)

fmt.Println(hex.EncodeToString(generated))

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
789cf248cdc9c95708cf2fca4901040000ffff180b041d
true
[{"name":"zlibCodec","params":"{\"mode\":-1}"}]
Example (Mode) 
data := []byte("Hello World")

c, _ := NewZlib(WithCompressionMode(BestSpeed))

generated, _ := c.Encode(data)

fmt.Println(hex.EncodeToString(generated))

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
7801000b00f4ff48656c6c6f20576f726c64010000ffff180b041d
true
[{"name":"zlibCodec","params":"{\"mode\":1}"}]

func NewZlibWithParent 

func NewZlibWithParent(parent Codec, opts ...func(*ZlibCodecOptions)) (Codec, error)

NewZlibWithParent returns a Codec that has at least one precursor codec, which will return compressed data

Example (Mode) 
data := []byte("Hello World")

parent, _ := NewReverser()

c, _ := NewZlibWithParent(parent, WithCompressionMode(BestSpeed))

generated, _ := c.Encode(data)

fmt.Println(hex.EncodeToString(generated))

rollback, _ := c.Decode(generated)

fmt.Println(bytes.Equal(data, rollback))
fmt.Println(c.Pipeline())

Output:
7801000b00f4ff646c726f57206f6c6c6548010000ffff195b041d
true
[{"name":"reverserCodec","params":""},{"name":"zlibCodec","params":"{\"mode\":1}"}]

func ParseEncrypter 

func ParseEncrypter(params []byte, parent Codec, opts map[string]any) (Codec, error)

ParseEncrypter creates a Encypter Codec with specified params and parent

func ParseNoOp 

func ParseNoOp(_ []byte, parent Codec, _ map[string]any) (Codec, error)

ParseNoOp creates a NoOp Codec with specified parent (params are ignored)

func ParseReverser 

func ParseReverser(_ []byte, parent Codec, _ map[string]any) (Codec, error)

ParseReverser creates a Reverser Codec with specified parent (params are ignored)

func ParseZlib 

func ParseZlib(params []byte, parent Codec, _ map[string]any) (Codec, error)

ParseZlib creates a Zlib Codec with specified params and parent

type CodecParser 

type CodecParser func(param []byte, parent Codec, opts map[string]any) (Codec, error)

CodecParser is the type of all Codec parsers, creating a new instance of Codec based on the specified parameters and parent (if specffied)

type EncrypterCodecOptions 

type EncrypterCodecOptions struct {
	PasswordTag string `json:"ptag"`
}

EncryptCodecOptions allows compression to be configured

func (*EncrypterCodecOptions) String 

func (e *EncrypterCodecOptions) String() string

String returns a stringised representation of the options

type Meta 

type Meta struct {
	// Version identifies the version of data as it is able to be deserialised, allowing correct unpacking
	Version string
	// BatchID allows identification of the overall processing
	BatchID []byte
	// SegmentID identifies what this specific piece of data is within the overall processing
	SegmentID []byte
	// Nonce ensures that this Package is uniquely identifiable across all processing
	Nonce []byte
	// CreationUTCTimestamp describes the UTC time when the Package was created
	CreationUTCTimestamp time.Time
	// Pipeline describes the Codec used to create the data being passed
	Pipeline []byte
	// ChainHash is SHA-512(priorChainHash || SHA-512(data)), linking this segment
	// to its predecessor to detect tampering and incorrect reassembly order
	ChainHash []byte
}

Meta describes what the data is, and how it is packaged

func (*Meta) Bytes 

func (m *Meta) Bytes() []byte

Bytes returns a dot "." delimited byte slice containing the meta fields (excluding version), where the constituents are ordered per the version of Packaging used and are base64 encoded

type Package 

type Package struct {
	// Meta provides information on what the data represents and how it can be recovered
	Meta *Meta
	// Data is the information being transferred
	Data []byte
	// Signature stores the signature of the creator, which should be verified by the recipient against tampering
	Signature []byte
}

Package captures all the necessary details that describe the underlying data and allow it to be unpackaged successfully

func CreatePackage 

func CreatePackage(b []byte, c Codec, batchID BatchID, segmentID SegmentID, privKey ed25519.PrivateKey, priorChainHash []byte) (*Package, error)

CreatePackage returns a Package that can be serialised to a byte slice for transmission.

Package instances contain metadata that describe what the data represents, and sufficient information to allow unpacking to the original data. Package instances are signed using the specific privKey to allow recipients to verify that it has not been modified during transmission.

The priorChainHash parameter enables chained hash verification across an orderd sequence of related Packages. It should be nil for the first Package, and the ChainHash from the preceding Package for the next Package in the sequence. The resulting ChainHash is stored in the Package's Meta and included in the Package signature. This provides a guarantee that the consumption of Packages is correctly ordered, and further restricts the ability to tamper with any one Package in the sequence.

func UnpackPackage 

func UnpackPackage(b []byte, pubKey ed25519.PublicKey) (*Package, error)

UnpackPackage reconstructs a Package from its serialised byte slice representation and verifies its signature.

The input bytes are expected to be from a prior call to Package.Bytes(). The public key must correspond to the private key used when the Package was created. Returns an error if the bytes cannot be parsed, the version is unsupported, or signature verification fails.

func (*Package) Bytes 

func (p *Package) Bytes() []byte

Bytes returns a dot "." delimited byte slice in the format: version.meta.data.signature where meta, data, and signature are base64 encoded. The meta field internally contains dot-delimited base64-encoded fields as defined by the version.

func (*Package) CodecFromPipeline 

func (p *Package) CodecFromPipeline(opts map[string]any) (Codec, error)

CodecFromPipeline returns a Codec instance that has been reconstructed from the Pipeline definition in the Package.Meta

type PasswordRetriever 

type PasswordRetriever func(passwordTag string) ([]byte, error)

PasswordRetriever is a function that can return the appropriate password for the specified tag The password length must be 32 bytes (i.e. for AES-256)

type Pipeline 

type Pipeline []PipelineElement

Pipeline is an ordered sequence of PipelineElements

func ParsePipeline 

func ParsePipeline(b []byte) (Pipeline, error)

ParsePipeline attempts to deserialise the bytes to a Pipeline of PipelineElements

func (Pipeline) Bytes 

func (p Pipeline) Bytes() ([]byte, error)

Bytes returns the sequence as a byte slice

func (Pipeline) String 

func (p Pipeline) String() string

String is the stringised representation of the Pipeline

type PipelineElement 

type PipelineElement struct {
	Name   string `json:"name"`
	Params string `json:"params"`
}

PipelineElement captures the non-secret details of a given Codec in a processing Pipeline

type SegmentID 

type SegmentID []byte

SegmentID is an identifier for the specific data being packaged, allowing successful reconstruction regardless of transmission sequence

type ZlibCodecOptions 

type ZlibCodecOptions struct {
	Compression int `json:"mode"`
}

ZlibCodecOptions allows compression to be configured

func (*ZlibCodecOptions) String 

func (z *ZlibCodecOptions) String() string

String returns a stringised representation of the options

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.