jsontrim

README

jsontrim

A lightweight, high-performance Go library for trimming JSON payloads to enforce size limits while preserving validity and data structure. Perfect for API responses, structured logging, and event recording where oversized data can cause outages.

Key features:

• Smart Limits: Drop or truncate fields > N bytes recursively.
• Total Size Cap: Iteratively remove elements until under the limit using size estimation to reduce GC pressure.
• Wildcard Blacklisting: Exclude sensitive paths dynamically (e.g., users.*.password).
• Ghost Markers: Optionally replace dropped fields with "[TRIMMED]" instead of deleting them, preserving schema visibility.
• Order Preservation: Safely trims arrays without destroying element order.
• Strategies: Choose removal order (largest-first, FIFO, prioritize keys).
• Hooks: Custom pre/post processing.
• Zero dependencies beyond encoding/json.

Installation

go get github.com/arun0009/jsontrim

Quick Start

package main

import (
	"fmt"
    "strings"

	"github.com/arun0009/jsontrim"
)

func main() {
    // A large JSON payload with sensitive data
	raw := []byte(`{
        "id": "123", 
        "data": "` + strings.Repeat("x", 2000) + `", 
        "users": [{"id":1, "pass":"secret"}, {"id":2, "pass":"secret"}]
    }`)

	trimmer := jsontrim.New(jsontrim.Config{
		FieldLimit:        500,
		TotalLimit:        1024,
		Blacklist:         []string{"users.*.pass"}, // Wildcard support
		ReplaceWithMarker: true,                     // Leave a trace of what was removed
		Strategy:          jsontrim.RemoveLargest{},
	})

	out, err := trimmer.Trim(raw)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Trimmed JSON (%d bytes): %s\n", len(out), out)
	// Output: {"id":"123","data":"[TRIMMED]","users":[{"id":1},{"id":2}]}
}

Configuration

Pass a Config to New():

• FieldLimit (int, default: 500): Max bytes per field/object/array (after nested trim).
• TotalLimit (int, default: 1024): Max total output bytes.
• Blacklist ([]string, default: []): Dot-notation paths to exclude. Supports * wildcards.
• ReplaceWithMarker (bool, default: false): If true, removed fields/items are replaced with "[TRIMMED]" string value instead of being deleted. Useful for debugging.
• Strategy (TruncStrategy, default: RemoveLargest): Removal policy: RemoveLargest{}, FIFO{}, or PrioritizeKeys.
• MaxDepth (int, default: 10): Recursion depth to prevent stack overflows.
• TruncateStrings (bool, default: false): Append "..." to oversized strings instead of dropping.
• Hooks (Hooks, default: {}): PreTrim/PostTrim funcs for custom logic.

Strategies

• RemoveLargest{}: Greedily drops the biggest fields/items to maximize retention (default).
• FIFO{}: Removes in iteration order (faster for ordered data).
• PrioritizeKeys{KeepKeys: []string{"id", "ts"}, Fallback: &FIFO{}}: Delays removal of key fields.

Blacklisting & Wildcards

Uses dot-notation. Supports * as a wildcard for array indices or dynamic map keys.

• "user.password": Matches exact path.
• "users.*.password": Matches password inside any element in users (e.g., array index users[0].password or map key users.primary.password).
• "logs.*": Matches everything inside logs.

Hooks Example

import (
	"log"
	"time"  
)

Hooks{
    PreTrim: func(v interface{}) interface{} {
        if m, ok := v.(map[string]interface{}); ok {
            m["trimmed_at"] = time.Now().Format(time.RFC3339)
        }
        return v
    },
    PostTrim: func(v interface{}, err error) interface{} {
        if err != nil {
            log.Printf("Trim error: %v", err)
        }
        return v
    },
}

Use Cases

• Structured Logging: Prevent large fields (like massive stack traces, base64 images, or entire HTTP bodies) from crashing log aggregators (ELK, Splunk) or consuming excessive bandwidth. jsontrim acts as a safety valve in log hooks.
• API Middleware: Ensure API responses strictly adhere to size contracts, preventing issues in client-side applications or with platform limits (e.g., Lambda/API Gateway payload size caps).
• Audit and Compiance: Use the Blacklist (especially with wildcards) to sanitize Personally Identifiable Information (PII) or sensitive credentials before persisting the data to an audit log or database.
• Event Stream Processing: Guarantee that messages pushed to queues (Kafka, RabbitMQ, SQS) never exceed topic size limits, avoiding message rejection and processing failures.

License

MIT License. See LICENSE.

Documentation

Index

• Variables
• type Config
• type FIFO
  • func (s FIFO) SelectNextToRemove(v interface{}) string
• type Hooks
• type PrioritizeKeys
  • func (s PrioritizeKeys) SelectNextToRemove(v interface{}) string
• type RemoveLargest
  • func (s RemoveLargest) SelectNextToRemove(v interface{}) string
• type Trimmer
  • func New(cfg Config) *Trimmer
  • func (t *Trimmer) Trim(raw []byte) ([]byte, error)
• type TruncStrategy

Variables

var (
	// ErrCannotTrim indicates the JSON couldn't be reduced below limits.
	ErrCannotTrim = errors.New("cannot trim JSON below limits")
	// Marker is the value used when ReplaceWithMarker is true.
	Marker = "[TRIMMED]"
)

Types

type Config

type Config struct {
	FieldLimit        int           // Max bytes per field/object/array (default: 500)
	TotalLimit        int           // Max total output bytes (default: 1024)
	Blacklist         []string      // Paths to exclude. Supports wildcards (e.g., "users.*.email")
	Strategy          TruncStrategy // Removal order during total enforcement (default: RemoveLargest)
	MaxDepth          int           // Recursion depth limit (default: 10)
	TruncateStrings   bool          // Truncate long strings with "..." instead of dropping (default: false)
	ReplaceWithMarker bool          // If true, replaced fields become "[TRIMMED]" instead of being deleted
	Hooks             Hooks         // Optional pre/post callbacks
}

Config holds customization options for the Trimmer.

type FIFO

type FIFO struct{}

Built-in strategies.

func (FIFO) SelectNextToRemove

func (s FIFO) SelectNextToRemove(v interface{}) string

SelectNextToRemove for FIFO: First key or index 0.

type Hooks

type Hooks struct {
	PreTrim  func(v interface{}) interface{}
	PostTrim func(v interface{}, err error) interface{}
}

Hooks for extensibility.

type PrioritizeKeys

type PrioritizeKeys struct {
	KeepKeys []string
	Fallback TruncStrategy
}

Built-in strategies.

func (PrioritizeKeys) SelectNextToRemove

func (s PrioritizeKeys) SelectNextToRemove(v interface{}) string

SelectNextToRemove for PrioritizeKeys: Skips keep keys, falls back.

type RemoveLargest

type RemoveLargest struct{}

Built-in strategies.

func (RemoveLargest) SelectNextToRemove

func (s RemoveLargest) SelectNextToRemove(v interface{}) string

SelectNextToRemove for RemoveLargest: Finds the largest by approximate size.

type Trimmer

type Trimmer struct {
	// contains filtered or unexported fields
}

Trimmer is the main struct.

func New

func New(cfg Config) *Trimmer

New creates a Trimmer with defaults filled.

func (*Trimmer) Trim

func (t *Trimmer) Trim(raw []byte) ([]byte, error)

Trim takes raw JSON bytes, strips blacklist, applies limits, and returns trimmed bytes.

type TruncStrategy

type TruncStrategy interface {
	// SelectNextToRemove identifies the next field/item to drop.
	// Returns key (map) or "idx:N" (array). Empty string if done.
	SelectNextToRemove(v interface{}) string
}

TruncStrategy defines removal policies for EnforceTotalLimit.