scanner

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 4, 2020 License: MIT Imports: 7 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsRegexOp

func IsRegexOp(t Token) bool

IsRegexOp returns true if the operator accepts a regex operand.

func ScanBareIdent

func ScanBareIdent(r io.RuneScanner) string

ScanBareIdent reads bare identifier from a rune reader.

func ScanDelimited

func ScanDelimited(r io.RuneScanner, start, end rune, escapes map[rune]rune, escapesPassThru bool) ([]byte, error)

ScanDelimited reads a delimited set of runes

func ScanString

func ScanString(r io.RuneReader) (string, error)

ScanString reads a quoted string from a rune reader.

func Tokstr

func Tokstr(tok Token, lit string) string

Tokstr returns a literal if provided, otherwise returns the token string.

Types

type BufScanner

type BufScanner struct {
	// contains filtered or unexported fields
}

BufScanner represents a wrapper for scanner to add a buffer. It provides a fixed-length circular buffer that can be unread.

func NewBufScanner

func NewBufScanner(r io.Reader) *BufScanner

NewBufScanner returns a new buffered scanner for a reader.

func (*BufScanner) Curr added in v0.5.0

func (s *BufScanner) Curr() TokenInfo

Curr returns the last read token.

func (*BufScanner) Scan

func (s *BufScanner) Scan() TokenInfo

Scan reads the next token from the scanner.

func (*BufScanner) ScanRegex

func (s *BufScanner) ScanRegex() TokenInfo

ScanRegex reads a regex token from the scanner.

func (*BufScanner) Unscan

func (s *BufScanner) Unscan()

Unscan pushes the previously token back onto the buffer.

type Pos

type Pos struct {
	Line int
	Char int
}

Pos specifies the line and character position of a token. The Char and Line are both zero-based indexes.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner represents a lexical scanner for Genji.

func NewScanner

func NewScanner(r io.Reader) *Scanner

NewScanner returns a new instance of Scanner.

func (*Scanner) ReadRune added in v0.5.0

func (s *Scanner) ReadRune() (ch rune, size int, err error)

func (*Scanner) Scan

func (s *Scanner) Scan() TokenInfo

Scan returns the next token and position from the underlying reader. Also returns the literal text read for strings, numbers, and duration tokens since these token types can have different literal representations.

func (*Scanner) ScanRegex

func (s *Scanner) ScanRegex() TokenInfo

ScanRegex consumes a token to find escapes

type Token

type Token int

Token is a lexical token of the Genji SQL language.

const (
	// ILLEGAL Token, EOF, WS are Special Genji SQL tokens.
	ILLEGAL Token = iota
	EOF
	WS
	COMMENT

	// IDENT and the following are Genji SQL literal tokens.
	IDENT           // main
	NAMEDPARAM      // $param
	POSITIONALPARAM // ?
	NUMBER          // 12345.67
	INTEGER         // 12345
	DURATION        // 13h
	STRING          // "abc"
	BADSTRING       // "abc
	BADESCAPE       // \q
	TRUE            // true
	FALSE           // false
	NULL            // NULL
	REGEX           // Regular expressions
	BADREGEX        // `.*

	// ADD and the following are Genji SQL Operators
	ADD        // +
	SUB        // -
	MUL        // *
	DIV        // /
	MOD        // %
	BITWISEAND // &
	BITWISEOR  // |
	BITWISEXOR // ^

	AND // AND
	OR  // OR

	EQ       // =
	NEQ      // !=
	EQREGEX  // =~
	NEQREGEX // !~
	LT       // <
	LTE      // <=
	GT       // >
	GTE      // >=

	LPAREN      // (
	RPAREN      // )
	LBRACKET    // {
	RBRACKET    // }
	LSBRACKET   // [
	RSBRACKET   // ]
	COMMA       // ,
	COLON       // :
	DOUBLECOLON // ::
	SEMICOLON   // ;
	DOT         // .

	// ALL and the following are Genji SQL Keywords
	AS
	ASC
	BY
	CAST
	CREATE
	DELETE
	DESC
	DROP
	EXISTS
	FROM
	IF
	INDEX
	INSERT
	INTO
	KEY
	LIMIT
	NOT
	OFFSET
	ON
	ORDER
	PRIMARY
	SELECT
	SET
	TABLE
	TO
	UNIQUE
	UPDATE
	VALUES
	WHERE

	TYPEBYTES
	TYPESTRING
	TYPEBOOL
	TYPEINT8
	TYPEINT16
	TYPEINT32
	TYPEINT64
	TYPEINT
	TYPEFLOAT64
	TYPEDURATION
	TYPEINTEGER // alias to TYPEINT
	TYPENUMERIC // alias to TYPEFLOAT64
	TYPETEXT    // alias to TYPESTRING

)

These are a comprehensive list of Genji SQL language tokens.

func Lookup

func Lookup(ident string) Token

Lookup returns the token associated with a given string.

func (Token) IsOperator

func (tok Token) IsOperator() bool

IsOperator returns true for operator tokens.

func (Token) Precedence

func (tok Token) Precedence() int

Precedence returns the operator precedence of the binary operator token.

func (Token) String

func (tok Token) String() string

String returns the string representation of the token.

type TokenInfo added in v0.5.0

type TokenInfo struct {
	Tok Token
	Pos Pos
	Lit string
	Raw string
}

TokenInfo holds information about a token.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL