dumper

README

Dumper Module

The dumper module provides functionality to dump data to files when errors occur. It supports dumping generic structs as JSON, stream events as JSONL (JSON Lines), and raw byte data as binary files.

Configuration

Add the following to your config.yml to enable and configure the dumper:

dumper:
  enabled: true                 # Enable data dumping on errors
  dump_path: "./dumps"          # Directory to dump data to
  max_size: 100                 # Maximum size of dump files in MB
  max_age: "24h"                # Maximum age of dump files to keep
  max_backups: 10               # Maximum number of old dump files to retain

Or use environment variables:

export LLM_PROXY_DUMPER_ENABLED=true
export LLM_PROXY_DUMPER_DUMP_PATH="./dumps"
export LLM_PROXY_DUMPER_MAX_SIZE=100
export LLM_PROXY_DUMPER_MAX_AGE="24h"
export LLM_PROXY_DUMPER_MAX_BACKUPS=10

Usage

Creating a Dumper Instance

// Create a logger
logger := log.New(log.Config{
    Level: 0, // Debug level
})

// Create dumper config
config := dumper.Config{
    Enabled:    true,
    DumpPath:   "./dumps",
    MaxSize:    100,
    MaxAge:     24 * time.Hour,
    MaxBackups: 10,
}

// Create a dumper
dumper := dumper.New(config, logger)

Dumping Structs as JSON

// Assuming you have a dumper instance
err := dumper.DumpStruct(context, myDataStruct, "error_data")
if err != nil {
    // Handle error
}

Dumping Stream Events as JSONL

// Assuming you have a dumper instance and a slice of events
events := []interface{}{
    map[string]interface{}{"event": "start", "timestamp": time.Now().Unix()},
    map[string]interface{}{"event": "process", "data": "processing user data"},
    map[string]interface{}{"event": "complete", "result": "success"},
}

err := dumper.DumpStreamEvents(context, events, "stream_events")
if err != nil {
    // Handle error
}

Dumping Raw Bytes

// Assuming you have a dumper instance and some byte data
byteData := []byte{0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64}

err := dumper.DumpBytes(context, byteData, "raw_data")
if err != nil {
    // Handle error
}

File Naming

Dumped files are named with the pattern: {filename}_{timestamp}.{extension}

• For structs: {filename}_{YYYYMMDD_HHMMSS}.json
• For stream events: {filename}_{YYYYMMDD_HHMMSS}.jsonl
• For raw bytes: {filename}_{YYYYMMDD_HHMMSS}.bin

Notes

• The dumper is thread-safe
• When disabled, dump operations return nil without error
• The dump directory is created automatically if it doesn't exist

Documentation

Overview

Package dumper for internal debug use only.

Index

• func DumpBytes(ctx context.Context, data []byte, filename string)
• func DumpObject(ctx context.Context, obj any, filename string)
• func DumpStreamEvents(ctx context.Context, events []*httpclient.StreamEvent, filename string)
• func Enabled() bool
• type Config
  • func DefaultConfig() Config
• type Dumper
  • func New(config Config) *Dumper
  • func (d *Dumper) DumpBytes(ctx context.Context, data []byte, filename string)
  • func (d *Dumper) DumpStreamEvents(ctx context.Context, events []*httpclient.StreamEvent, filename string)
  • func (d *Dumper) DumpStruct(ctx context.Context, data any, filename string)

Functions

func DumpBytes

func DumpBytes(ctx context.Context, data []byte, filename string)

func DumpObject

func DumpObject(ctx context.Context, obj any, filename string)

func DumpStreamEvents

func DumpStreamEvents(ctx context.Context, events []*httpclient.StreamEvent, filename string)

func Enabled

func Enabled() bool

Types

type Config

type Config struct {
	// Enabled enables or disables the dumper.
	Enabled bool `conf:"enabled" yaml:"enabled" json:"enabled"`

	// DumpPath is the directory path where data will be dumped.
	DumpPath string `conf:"dump_path" yaml:"dump_path" json:"dump_path"`

	// MaxSize is the maximum size of dump files in MB.
	MaxSize int `conf:"max_size" yaml:"max_size" json:"max_size"`

	// MaxAge is the maximum age of dump files to keep.
	MaxAge time.Duration `conf:"max_age" yaml:"max_age" json:"max_age"`

	// MaxBackups is the maximum number of old dump files to retain.
	MaxBackups int `conf:"max_backups" yaml:"max_backups" json:"max_backups"`
}

Config holds the configuration for the dumper.

func DefaultConfig

func DefaultConfig() Config

type Dumper

type Dumper struct {
	// contains filtered or unexported fields
}

Dumper is responsible for dumping data to files when errors occur.

func New

func New(config Config) *Dumper

New creates a new Dumper instance.

func (*Dumper) DumpBytes

func (d *Dumper) DumpBytes(ctx context.Context, data []byte, filename string)

DumpBytes dumps raw byte data to a file.

func (*Dumper) DumpStreamEvents

func (d *Dumper) DumpStreamEvents(ctx context.Context, events []*httpclient.StreamEvent, filename string)

DumpStreamEvents dumps a slice of interface{} as JSONL (JSON Lines) to a file.

func (*Dumper) DumpStruct

func (d *Dumper) DumpStruct(ctx context.Context, data any, filename string)

DumpStruct dumps any struct as JSON to a file.