lexer

Documentation

Overview

Package lexer contains the code to lex input programs into a stream of tokens. The stream of tokens can then be parsed into an AST.

Index

• type Lexer
  • func New(input string, options ...Option) *Lexer
  • func (l *Lexer) Filename() string
  • func (l *Lexer) GetLineText(t token.Token) string
  • func (l *Lexer) Next() (token.Token, error)
  • func (l *Lexer) Position() token.Position
  • func (l *Lexer) SetFilename(file string)
• type NumberType
• type Option
  • func WithFile(file string) Option

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

Lexer holds our object-state.

func New

func New(input string, options ...Option) *Lexer

New returns a Lexer instance for the given string input.

func (*Lexer) Filename

func (l *Lexer) Filename() string

Filename returns the name of the file being read.

func (*Lexer) GetLineText

func (l *Lexer) GetLineText(t token.Token) string

GetLineText returns the text of the line containing the given token.

func (*Lexer) Next

func (l *Lexer) Next() (token.Token, error)

Next returns the next Token from the input that is being lexed.

func (*Lexer) Position

func (l *Lexer) Position() token.Position

Position returns the current read position of the Lexer as a Position object.

func (*Lexer) SetFilename

func (l *Lexer) SetFilename(file string)

SetFilename sets the name of the file being read.

type NumberType

type NumberType string

NumberType describes the type of a number that is being lexed.

const (
	NumberTypeInvalid NumberType = "invalid"
	NumberTypeDecimal NumberType = "decimal"
	NumberTypeHex     NumberType = "hex"
	NumberTypeBinary  NumberType = "binary"
)

type Option

type Option func(*Lexer)

Option is a configuration function for a Lexer.

func WithFile

func WithFile(file string) Option

WithFile sets the file name for the Lexer.