jsonl

Documentation

Overview

Package jsonl provides line-oriented IO primitives for JSONL files that may be appended to while being read. It is provider-neutral: callers get raw lines and decide what to parse, so unmodeled entry shapes remain reachable without a library release.

The oversized-line rule is load-bearing: lines exceeding the max line size are silently skipped, never aborting iteration (unlike bufio.Scanner, whose ErrTooLong permanently kills a scan). The default max of 64 MiB pins tail-claude's semantics verbatim; callers with different needs supply their own via NewReaderSize.

Index

• Constants
• func ReverseScan(path string, maxBytes int64, fn func(line []byte) bool) error
• func ScanLines(r io.Reader, fn func(line string) bool) error
• func TailLines(path string, maxBytes int64) ([][]byte, error)
• type Reader
  • func NewReader(r io.Reader) *Reader
  • func NewReaderSize(r io.Reader, maxLineSize int) *Reader
  • func (lr *Reader) BytesRead() int64
  • func (lr *Reader) Err() error
  • func (lr *Reader) LastLineTerminated() bool
  • func (lr *Reader) Next() (string, bool)
  • func (lr *Reader) TerminatedBytesRead() int64

Examples

• NewReader (OffsetTracking)
• ScanLines

Constants

const (

	// DefaultMaxLineSize is the maximum allowed line length when the caller
	// does not supply one via NewReaderSize. Lines exceeding the limit are
	// silently skipped rather than aborting the entire scan. 64 MiB
	// accommodates even the largest Claude API responses (inline images,
	// giant pre-2.1.19x tool results).
	DefaultMaxLineSize = 64 * 1024 * 1024
)

Functions

func ReverseScan

func ReverseScan(path string, maxBytes int64, fn func(line []byte) bool) error

ReverseScan calls fn for each line in the last maxBytes of the file at path, newest first, stopping when fn returns false. The final line handed to fn (the oldest in the window) may be truncated when the window starts mid-file; callers must tolerate an unparseable segment.

func ScanLines

func ScanLines(r io.Reader, fn func(line string) bool) error

ScanLines calls fn for each non-empty line of r, stopping early when fn returns false. Lines exceeding DefaultMaxLineSize are skipped rather than aborting the scan — unlike bufio.Scanner, whose ErrTooLong permanently kills iteration, so one huge line (e.g. pasted image data) would hide every line after it. Returns the first I/O error encountered, or nil.

Example

package main

import (
	"fmt"
	"strings"

	"github.com/kylesnowschwartz/agent-ouija/jsonl"
)

func main() {
	data := "{\"a\":1}\n{\"b\":2}\n"
	_ = jsonl.ScanLines(strings.NewReader(data), func(line string) bool {
		fmt.Println(line)
		return true
	})
}

Output:
{"a":1}
{"b":2}

func TailLines

func TailLines(path string, maxBytes int64) ([][]byte, error)

TailLines reads at most maxBytes from the end of the file at path and returns the newline-split segments, oldest first. When the read window starts mid-file, the first segment is almost certainly a truncated line — callers must tolerate (or skip) an unparseable leading segment. Empty segments are omitted.

This is the bounded bottom-up scan pattern for huge transcripts: the interesting entry (e.g. the last assistant model) sits near the end of a file that may be hundreds of MB.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads JSONL files line by line, skipping lines that exceed the max line size rather than aborting. The buffer starts small and grows on demand. After iteration, call Err() to check for I/O errors (not EOF).

Ported from agentsview's internal/parser/linereader.go with the addition of BytesRead() for incremental offset tracking.

func NewReader

func NewReader(r io.Reader) *Reader

NewReader returns a Reader with the DefaultMaxLineSize limit.

Example (OffsetTracking)

package main

import (
	"fmt"
	"strings"

	"github.com/kylesnowschwartz/agent-ouija/jsonl"
)

func main() {
	r := jsonl.NewReader(strings.NewReader("{\"a\":1}\n{\"partial\":"))
	for {
		if _, ok := r.Next(); !ok {
			break
		}
	}
	// TerminatedBytesRead excludes the unterminated tail, so an
	// incremental reader can resume without splitting the half-written line.
	fmt.Println(r.TerminatedBytesRead() < r.BytesRead())
}

Output:
true

func NewReaderSize

func NewReaderSize(r io.Reader, maxLineSize int) *Reader

NewReaderSize returns a Reader that skips lines longer than maxLineSize bytes. A maxLineSize of 0 (or less) means DefaultMaxLineSize.

func (*Reader) BytesRead

func (lr *Reader) BytesRead() int64

BytesRead returns the total bytes consumed from the reader, including skipped lines, newline delimiters, and any EOF-truncated trailing line.

func (*Reader) Err

func (lr *Reader) Err() error

Err returns the first non-EOF I/O error encountered, or nil.

func (*Reader) LastLineTerminated

func (lr *Reader) LastLineTerminated() bool

LastLineTerminated reports whether the most recently returned line ended with a newline. False means the line was cut off at EOF -- during live tailing that is typically an append still in progress.

func (*Reader) Next

func (lr *Reader) Next() (string, bool)

Next returns the next non-empty line (without trailing newline) and true, or ("", false) at EOF or I/O error. After the loop, call Err() to distinguish EOF from I/O failure.

func (*Reader) TerminatedBytesRead

func (lr *Reader) TerminatedBytesRead() int64

TerminatedBytesRead returns the bytes consumed through the last newline-terminated line, excluding an EOF-truncated tail. Used by ReadSessionIncremental for offset tracking during live tailing: a half-written trailing line must be re-read intact on the next call.