lexer

Documentation

Index

• type Lexer
  • func New(input string) *Lexer
  • func (l *Lexer) NextToken() token.Token

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer converts a string input to produce tokens for the Monkey language. It always keeps track of the current position, the next readable position and the current character under examination. These tokens will be parsed by the parser, which constructs the abstract syntax-tree (AST).

func New

func New(input string) *Lexer

New creates a new Lexer for a given input It calls readChar a single time to initialize the first char to be examined, then sets the position and the next readPosition for the lexer

func (*Lexer) NextToken

func (l *Lexer) NextToken() token.Token

NextToken looks at the current character under examination and returns a Token depending on which character it is.