jsont

README

jsont

A JSON tokenizer that also emits insignificant tokens like whitespaces.

{ "Hello" : "World" }

Will be split into the following tokens:

|{| |"Hello"| |:| |"World"| |}|

Usage

tokenizer := jsont.NewTokenizer(strings.NewReader(jsonInput))

for tokenizer.Next() {
	token := tokenizer.Token()
	fmt.Println(token.Type)
	fmt.Println(token.Value)
}                                                                                                                                   

if tokenizer.Error() != nil {
	fmt.Println(tokenizer.Error())
}

Disclaimer

• Only consider using this tokenizer if you are intersted in whitespaces for some reason.
  Otherwise you should have a look at Go 1.5 which has a json tokenizer built in.
  • https://github.com/golang/go/issues/6050
  • https://github.com/golang/go/commit/0cf48b4d919e5137ec03f1bf230fbd8720873151
• Also notice that this tokenizer does not verify if the incoming data is valid JSON or not.

Documentation

Overview

Package jsont provides a non validating json tokenizer that also emits insignificant tokens (like whitespace).

Example

The following json:

{ "Hello" : "World" }

Will be split into the following tokens:

|{| |"Hello"| |:| |"World"| |}|

Usage

Usage example:

tokenizer := jsont.NewTokenizer(strings.NewReader(jsonInput))

for tokenizer.Next() {
	token := tokenizer.Token()
	fmt.Println(token.Type)
	fmt.Println(token.Value)
}

if tokenizer.Error() != nil {
	fmt.Println(tokenizer.Error())
}

Disclaimer

The behavior of this tokenizer is rarely useful. In fact only consider using it, if you are interested in whitespace tokens and don't need to check if the given input is valid json.

Index

• Constants
• type Token
• type TokenType
• type Tokenizer
  • func NewTokenizer(reader io.Reader) Tokenizer

Constants

const (
	Empty = iota
	Whitespace
	ObjectStart
	ObjectEnd
	ArrayStart
	ArrayEnd
	Colon
	Comma
	FieldName
	True
	False
	Null
	Integer
	Float
	String
	Unknown
)

Types

type Token

type Token struct {
	Type  TokenType
	Value string
}

type TokenType

type TokenType int

type Tokenizer

type Tokenizer interface {

	// Scan advances the Scanner to the next token, which will then be available
	// through the Token method. It returns false if no more Tokens are available,
	// either because of reaching the end of the input or an error.
	Next() bool

	// Token returns the most recent token generated by a call to Next.
	Token() Token

	// Error returns the first non-EOF error that was encountered by the Tokenizer.
	Error() error
}

func NewTokenizer

func NewTokenizer(reader io.Reader) Tokenizer