ndjson

Documentation

Overview

Package ndjson provides reading and appending for newline-delimited JSON (NDJSON/JSONL) files.

Wallfacer uses NDJSON for event sourcing trace files and turn-usage records where each line is a self-contained JSON object. This package consolidates the repeated pattern of opening a file, scanning lines, and unmarshaling JSON into generic ReadFile and ReadFileFunc functions, plus an atomic AppendFile for concurrent-safe record appending. Missing files are treated as empty (not errors).

Connected packages

No internal dependencies (stdlib only). Consumed by [store] for reading and writing task event traces, turn usage records, and ideation history. Changes to the file format or error handling affect all event sourcing persistence.

Usage

events, err := ndjson.ReadFile[Event](tracePath)
err = ndjson.AppendFile(tracePath, newEvent)
err = ndjson.ReadFileFunc(path, func(e Event) bool {
    return e.Type == "state_change" // stop iteration on false
})

Package ndjson provides helpers for reading and appending newline-delimited JSON (NDJSON/JSONL) files. It consolidates the repeated scanner-based read/append patterns found throughout the codebase.

Index

• func AppendFile[T any](path string, record T) error
• func ReadFile[T any](path string, opts ...Option) ([]T, error)
• func ReadFileFunc[T any](path string, fn func(T) bool, opts ...Option) error
• type Option
  • func WithBufferSize(initial, maxSize int) Option
  • func WithOnError(fn func(lineNum int, err error)) Option

Functions

func AppendFile

func AppendFile[T any](path string, record T) error

AppendFile atomically appends a single JSON-encoded record followed by a newline to path. The file is created if it does not exist. The write uses O_APPEND so concurrent appends of records under 4 KB are atomic on Linux.

func ReadFile

func ReadFile[T any](path string, opts ...Option) ([]T, error)

ReadFile opens path, decodes each JSON line into T, and returns the collected results. Empty and whitespace-only lines are skipped.

If path does not exist, ReadFile returns (empty non-nil slice, nil).

func ReadFileFunc

func ReadFileFunc[T any](path string, fn func(T) bool, opts ...Option) error

ReadFileFunc opens path and decodes each JSON line into T, calling fn for every successfully decoded record. Return false from fn to stop iteration early. Empty and whitespace-only lines are skipped.

If path does not exist, ReadFileFunc returns nil (no error).

Types

type Option

type Option func(*config)

Option configures NDJSON scanning behavior.

func WithBufferSize

func WithBufferSize(initial, maxSize int) Option

WithBufferSize sets the initial and maximum scanner buffer sizes. By default the bufio.Scanner default (64 KB max) is used. Use this when lines may exceed that limit.

func WithOnError

func WithOnError(fn func(lineNum int, err error)) Option

WithOnError sets a callback invoked for lines that fail to unmarshal. The callback receives the 1-based line number and the unmarshal error. By default malformed lines are silently skipped.