Documentation
¶
Overview ¶
Package token defines the JSON token types with their kind.
These data structures should be returned by all implementations of a JSON lexer.
There are two flavors of supported tokens:
- the basic token type T, which is suitable for most purposes
The verbatim token type should be reserved to specific use-cases such as rendering verbatim documents (e.g. for linters, auto-fixers, editor plugins, etc.)
Index ¶
- Variables
- func Unescape(raw []byte) []byte
- func UnescapeString(raw []byte) string
- type Kind
- type KindDelimiter
- type T
- func (t T) Bool() bool
- func (t T) Clone() T
- func (t T) Delimiter() KindDelimiter
- func (t T) IsBool() bool
- func (t T) IsColon() bool
- func (t T) IsComma() bool
- func (t T) IsDelimiter() bool
- func (t T) IsEOF() bool
- func (t T) IsEndArray() bool
- func (t T) IsEndObject() bool
- func (t T) IsKey() bool
- func (t T) IsKnown() bool
- func (t T) IsNull() bool
- func (t T) IsScalar() bool
- func (t T) IsStartArray() bool
- func (t T) IsStartObject() bool
- func (t T) Kind() Kind
- func (t T) String() string
- func (t T) Value() []byte
Constants ¶
This section is empty.
Variables ¶
var EOFToken = T{ // contains filtered or unexported fields }
EOFToken is a preallocated placeholder returned whenever the lexer has reached the end of the input stream.
var None = T{ // contains filtered or unexported fields }
None is a preallocated placeholder for any invalid or unrecognized JSON token.
var NullToken = T{ // contains filtered or unexported fields }
NullToken is a preallocated placeholder for "null" tokens.
Functions ¶
func Unescape ¶
Unescape returns the decoded form of a raw string/Key value produced by the verbatim lexer [lexer.VL].
VL keeps string/Key values exactly as they appeared in the source (escape sequences intact — see the String/Key doc), so the token stream can be round-tripped byte-for-byte. Unescape expands the JSON escapes on demand: the shorthand escapes (\", \\, \/, \b, \f, \n, \r, \t) and \uXXXX sequences (surrogate pairs combined) become their UTF-8 bytes.
If raw contains no escape it is returned unchanged with no allocation; otherwise a fresh slice is returned. Do NOT call this on a semantic-lexer value — the semantic lexer [lexer.L] already decodes.
The escapes were validated when the token was scanned, so decoding cannot fail; a malformed sequence would have errored at scan time (any residual bad input is passed through rather than panicking).
func UnescapeString ¶
UnescapeString is Unescape as a string. It always allocates (the string header cannot alias the token's buffer, which is reused on the next token).
Types ¶
type Kind ¶
type Kind uint8
Kind of JSON token, i.e. either a delimiter, a string, a number, a boolean or null.
EOF is considered a special token that marks the end of a JSON stream.
Strings and numbers are not converted to go string and go numeric types respectively: the original value is kept as a slice of bytes.
const ( // Unknown token. // // This result is associated with an error in the lexer. Unknown Kind = iota // Delimiter token, i.e. ",", ":", "{", "}", "[", "]". Delimiter // String token. // // From the semantic lexer [lexer.L], [T.Value] is unescaped (decoded); from the // verbatim lexer [lexer.VL] it keeps the raw source bytes (escapes intact) for // faithful round-tripping — decode on demand with [Unescape]. String // Key string token. // // Like [String]: semantic value is decoded, verbatim value is raw (see [Unescape]). Key // Number JSON token. Number // Boolean token. Boolean // Null value token. Null // EOF signals that the lexer has reached the end of the input stream. EOF )
JSON tokens.
type KindDelimiter ¶
type KindDelimiter uint8
KindDelimiter represents a JSON delimiter (i.e. ":", ",", "{", "}", "[", "]").
const ( // NotADelimiter is the zero value, used when the token is not a delimiter. NotADelimiter KindDelimiter = iota // Comma is "," Comma // Colon is ":" Colon // OpeningBracket is "{" OpeningBracket // ClosingBracket is "}" ClosingBracket // OpeningSquareBracket is "[" OpeningSquareBracket // ClosingSquareBracket is "]" ClosingSquareBracket )
Delimiters.
func (KindDelimiter) AcceptValue ¶
func (d KindDelimiter) AcceptValue() bool
AcceptValue returns true when the delimiter may be followed by a value token.
Examples: ": true", "[\"abc\"]", ",123", {"abc" are legit but not: "} true", "] 123"
Notice that {123 or {true are accepted: more context is needed to reject such constructs.
func (KindDelimiter) IsClosing ¶
func (d KindDelimiter) IsClosing() bool
IsClosing returns true for closing delimiters such as "}" or "]"
func (KindDelimiter) String ¶
func (d KindDelimiter) String() string
String representation of a delimiter.
type T ¶
type T struct {
// contains filtered or unexported fields
}
T represents a JSON token.
Tokens are immutable, short-lived objects.
T maintains strings and numbers as slices of bytes representing an UTF8 string.
It doesn't keep track of non-significant blank space or new lines.
Escaped unicode sequences are unescaped as UTF8 runes.
Limitation: JSON data based on a non-UTF8 character set need to be converted beforehand.
func Make ¶
func Make(kind Kind, value []byte, delimiter KindDelimiter, valueBool bool) T
Make a token T.
func MakeDelimiter ¶
func MakeDelimiter(delimiter KindDelimiter) T
MakeDelimiter builds a delimiter token T.
func MakeWithValue ¶
MakeWithValue builds a scalar string or number token T.
func (T) Clone ¶
Clone deep-clones a token.
Memory to hold the token's string or numeric value will be freshly allocated.
func (T) Delimiter ¶
func (t T) Delimiter() KindDelimiter
Delimiter for delimiter tokens.
The value is NotADelimiter for non-delimiter tokens.
func (T) IsDelimiter ¶
func (T) IsEndArray ¶
func (T) IsEndObject ¶
func (T) IsScalar ¶
IsScalar indicates is the Token represents a scalar value (null is not considered a scalar).
Source Files