misc

Documentation

Index

• Constants
• type Lexer
  • func NewLexer(src *parser.SourceFile) *Lexer
  • func (lx *Lexer) Position() parser.Cursor
  • func (lx *Lexer) Read() (*parser.Token, error)
  • func (lx *Lexer) ReadExact(expectation string, kind parser.FragmentKind) (token *parser.Token, matched bool, err error)
  • func (lx *Lexer) Set(cursor parser.Cursor)

Constants

const (

	// FrSpace represents a space fragment kind
	FrSpace parser.FragmentKind

	// FrWord represents a word fragment kind
	FrWord

	// FrSign represents a special character fragment kind
	FrSign
)

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer represents a basic lexer tokenizing source code into 3 basic categories: spaces (whitespaces, tabs, line-breaks), signs (any ASCII special character) and words (any other character)

func NewLexer

func NewLexer(src *parser.SourceFile) *Lexer

NewLexer creates a new basic-latin lexer instance

func (*Lexer) Position

func (lx *Lexer) Position() parser.Cursor

Position returns the current position of the lexer

func (*Lexer) Read

func (lx *Lexer) Read() (*parser.Token, error)

Read returns either the next token or nil if end of file is reached

func (*Lexer) ReadExact

func (lx *Lexer) ReadExact(
	expectation string,
	kind parser.FragmentKind,
) (
	token *parser.Token,
	matched bool,
	err error,
)

ReadExact tries to read an exact string and returns false if str couldn't have been matched

func (*Lexer) Set

func (lx *Lexer) Set(cursor parser.Cursor)

Set sets a new cursor