token

Documentation

Overview

String quoting and unquoting for JSON. Unquote is what lets jsonsaw compare object keys against a --path without allocating per element; Quote is used when `jsonsaw join --path` has to synthesize wrapper keys.

Package token implements an incremental JSON tokenizer that reads its input in a single forward pass. Memory use is O(nesting depth + largest single token) — independent of document size — which is what lets jsonsaw saw through multi-gigabyte documents in a few megabytes of RAM.

The tokenizer is strict RFC 8259: it rejects leading zeros, trailing commas, bare words, unescaped control characters, and trailing data after the top-level value. A sequence mode (NewSequence) accepts a stream of whitespace-separated top-level values instead of exactly one, which is how jsonsaw reads JSONL without ever buffering a whole line.

Index

• Constants
• func Quote(s string) []byte
• func Unquote(raw []byte) (string, error)
• func ValueKind(k Kind) string
• type Kind
• type SyntaxError
  • func (e *SyntaxError) Error() string
• type Token
• type Tokenizer
  • func New(r io.Reader) *Tokenizer
  • func NewSequence(r io.Reader) *Tokenizer
  • func (t *Tokenizer) Depth() int
  • func (t *Tokenizer) Next() (Token, error)
  • func (t *Tokenizer) Pos() (line, col int)
  • func (t *Tokenizer) SetMaxDepth(n int)

Constants

const DefaultMaxDepth = 10000

DefaultMaxDepth caps container nesting so a hostile input of a billion open brackets cannot grow the state stack without bound.

Functions

func Quote

func Quote(s string) []byte

Quote encodes s as a JSON string, quotes included. Non-ASCII runes pass through as UTF-8; only quotes, backslashes, and control characters are escaped, so quoted keys stay human-readable.

func Unquote

func Unquote(raw []byte) (string, error)

Unquote decodes the raw bytes of a String or Key token (including the surrounding quotes) into the string it denotes. Escapes are resolved; lone UTF-16 surrogates become U+FFFD, matching encoding/json. The input is assumed to have passed the tokenizer, but malformed escapes still return an error rather than panic.

func ValueKind

func ValueKind(k Kind) string

ValueKind names the JSON type a token starts, for human-readable errors and for `jsonsaw paths` output.

Types

type Kind

type Kind int

Kind identifies a token. Structural commas and colons are consumed by the tokenizer itself and never surface; callers see only values, keys, and container boundaries.

const (
	BeginObject Kind = iota // {
	EndObject               // }
	BeginArray              // [
	EndArray                // ]
	Key                     // an object key (raw bytes include the quotes)
	String                  // a string value (raw bytes include the quotes)
	Number                  // a number value (raw source bytes)
	True                    // literal true
	False                   // literal false
	Null                    // literal null
)

type SyntaxError

type SyntaxError struct {
	Line int
	Col  int
	Msg  string
}

SyntaxError reports malformed JSON with the position of the offending byte. jsonsaw maps it to exit code 1.

func (*SyntaxError) Error

func (e *SyntaxError) Error() string

type Token

type Token struct {
	Kind  Kind
	Bytes []byte
	Line  int // 1-based line of the token's first byte
	Col   int // 1-based byte column of the token's first byte
}

Token is one lexical unit of the input.

Bytes holds the raw source bytes for Key, String, and Number tokens (strings keep their quotes and escapes exactly as written, so copying a token back out is byte-faithful). For True, False, and Null it holds the literal text. The slice aliases an internal scratch buffer and is only valid until the next call to Next; copy it if you need to keep it.

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer streams tokens from r. Create one with New or NewSequence.

func New

func New(r io.Reader) *Tokenizer

New returns a tokenizer for exactly one top-level JSON value. Trailing non-whitespace after the value is a SyntaxError.

func NewSequence

func NewSequence(r io.Reader) *Tokenizer

NewSequence returns a tokenizer that accepts any number of whitespace-separated top-level values (a superset of JSONL). Next returns io.EOF after the last value; an empty input is a zero-value sequence.

func (*Tokenizer) Depth

func (t *Tokenizer) Depth() int

Depth reports how many containers are currently open.

func (*Tokenizer) Next

func (t *Tokenizer) Next() (Token, error)

Next returns the next token. It returns io.EOF exactly once, at a clean end of input; any other error is sticky and repeats on later calls.

func (*Tokenizer) Pos

func (t *Tokenizer) Pos() (line, col int)

Pos reports the position of the next unread byte (1-based line and byte column).

func (*Tokenizer) SetMaxDepth

func (t *Tokenizer) SetMaxDepth(n int)

SetMaxDepth overrides DefaultMaxDepth. n must be at least 1.