scanner

Documentation

Overview

Package scanner tokenizes query strings.

Index

• type Scanner
  • func New(src io.Reader) Scanner
  • func (x *Scanner) Scan() (kind TokenKind, err error)
  • func (x *Scanner) Token() string
• type TokenKind

Types

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner splits the bytes from an io.Reader into tokens for a Parser. For each call to the Scan method, a token worth of bytes is read from the io.Reader and the kind of token is returned. After a call to Scan, the Token method may be called to obtain the token string.

func New

func New(src io.Reader) Scanner

New creates a Scanner which reads from the given io.Reader.

func (*Scanner) Scan

func (x *Scanner) Scan() (kind TokenKind, err error)

Scan reads a token from the wrapped io.Reader and returns the kind of token read. The Scanner is meant to work with any input and should never fail as long as the io.Reader doesn't fail. io.Reader errors are wrapped and returned by this method.

func (*Scanner) Token

func (x *Scanner) Token() string

Token returns token obtained from the wrapped io.Reader by the last call to Scan.

type TokenKind

type TokenKind int

TokenKind represents the kind of token read during a call to Scanner.Scan.

const (
	// TokenKindUnassigned is used to identify a TokenKind variable
	// which hasn't been assigned to yet. For this purpose, it must
	// be the zero-value of its type.
	TokenKindUnassigned TokenKind = iota

	// TokenKindWhitespace identifies a token which only contains
	// runes found in the runesWhitespace constant.
	TokenKindWhitespace

	// TokenKindNewline identifies a token which only contains
	// runes found in the runesWhitespace or runesNewline
	// constants.
	TokenKindNewline

	// TokenKindEscape identifies a 2-rune token which always
	// starts with the Escape rune.
	TokenKindEscape

	// TokenKindOther identifies all other possible tokens which are
	// not identified by the given TokenKind constants. This kind of
	// token is used to represent directory names, value types, and
	// data elements (numbers, strings, UUIDs, etc...).
	TokenKindOther

	// TokenKindEnd is returned from Scanner.Scan when the wrapped
	// io.Reader has been read to completion.
	TokenKindEnd

	// TokenKindKeyValSep identifies a token equal to KeyValSep.
	TokenKindKeyValSep

	// TokenKindDirSep identifies a token equal to DirSep.
	TokenKindDirSep

	// TokenKindTupStart identifies a token equal to TupStart.
	TokenKindTupStart

	// TokenKindTupEnd identifies a token equal to TupEnd.
	TokenKindTupEnd

	// TokenKindTupSep identifies a token equal to TupSep.
	TokenKindTupSep

	// TokenKindVarStart identifies a token equal to VarStart.
	TokenKindVarStart

	// TokenKindVarEnd identifies a token equal to VarEnd.
	TokenKindVarEnd

	// TokenKindVarSep identifies a token equal to VarSep.
	TokenKindVarSep

	// TokenKindStrMark identifies a token equal to StrMark.
	TokenKindStrMark

	// TokenKindStampStart identifies a token equal to StampStart.
	TokenKindStampStart

	// TokenKindStampSep identifies a token equal to StampSep.
	TokenKindStampSep

	// TokenKindReserved identifies a single-rune token which
	// isn't currently used by the language but reserved for
	// later use.
	TokenKindReserved
)