lexer

Documentation

Overview

================================================================================================= Alex Peters - January 22, 2024

errors specific to lexical analysis step =================================================================================================

============================================================================= Alex Peters - January 21, 2024 =============================================================================

================================================================================================= Modified source code from Go standard lib.

Original license located w/in the same directory as this file inside the file named GO-LICENSE

Modified by Alex Peters, January 29, 2024

See modification w/in function documentation

Lastly, any help on a better way to deal with this legal stuff would be greatly appreciated =================================================================================================

Index

• Constants
• func ScanLines(data []byte, atEOF bool) (advance int, token []byte, err error)
• type Lexer
  • func Init(path source.PathSpec) *Lexer
  • func (lex *Lexer) AddExtensionKey(x token.Token) bool
  • func (lex *Lexer) FlushMessages() []errors.ErrorMessage
  • func (lexer *Lexer) LinePos(line int) (start, end int)
  • func (lex *Lexer) Messages() []errors.ErrorMessage
  • func (lex *Lexer) Next() (tok token.Token, ok bool)
  • func (lex *Lexer) SetKeepComments(truthy bool)
  • func (lex *Lexer) SetSpaceWhitespace()
  • func (lex *Lexer) SetTabWhitespace()
  • func (lex *Lexer) String() string
  • func (lex *Lexer) Tokenize() (tokens []token.Token, ok bool)
  • func (lex *Lexer) ValidLine() bool
  • func (lex *Lexer) Write() int

Constants

const (
	InvalidCharacter                string = "invalid character"
	InvalidAffixId                  string = "invalid affixed identifier"
	InvalidCharacterAtEndOfNumConst string = "invalid character at end of numeric constant"
	UnexpectedEOF                   string = "unexpected end of file"
	ExpectedCharLiteral             string = "expected character literal"
	IllegalAffixedImplicitId        string = "illegal affixation of implicitly bindable identifier"
	IllegalCharLiteral              string = "illegal character literal"
	IllegalEscapeSequence           string = "illegal escape sequence"
	IllegalHoleId                   string = "illegal hole identifier"
	IllegalStringLiteral            string = "illegal string literal"
	IllegalUnderscoreSequence       string = "illegal contiguous underscores"
	IllegalWhitespace               string = "illegal whitespace"
	InvalidUnderscore               string = "invalid underscore"
	InvalidAnnotation               string = "invalid annotation"

	ExtensionOverwrite string = "keyword used in extension outside of an enclosing '\"' pair"
)

Functions

func ScanLines

func ScanLines(data []byte, atEOF bool) (advance int, token []byte, err error)

original documentation:

ScanLines is a split function for a Scanner that returns each line of text, stripped of any trailing end-of-line marker. The returned line may be empty. The end-of-line marker is one optional carriage return followed by one mandatory newline. In regular expression notation, it is `\r?\n`. The last non-empty line of input will be returned even if it has no newline.

modifications:

• within the second if statement's block, the line that read `return i + 1, dropCR(data[0:i]), nil` is modified to `return i + 1, dropCR(data[0 : i+1]), nil`

Types

type Lexer

type Lexer struct {
	source.SourceCode
	// current line number
	Line int
	// current position in source
	Pos int
	// saved char number
	SavedChar *stack.Stack[int]
	// Affixed Identifiers
	Affixed      []int
	AffixIndexes []int
	// tokens created from source
	Tokens []token.Token
	// contains filtered or unexported fields
}

func Init

func Init(path source.PathSpec) *Lexer

initialize lexer for writing to its internal source buffer `lex.Source` from the input stream located by path.

given

lex := lexer.Init(myPath)

write input to lex.Source with

lex.Write()

func (*Lexer) AddExtensionKey

func (lex *Lexer) AddExtensionKey(x token.Token) bool

func (*Lexer) FlushMessages

func (lex *Lexer) FlushMessages() []errors.ErrorMessage

func (*Lexer) LinePos

func (lexer *Lexer) LinePos(line int) (start, end int)

returns index position for given line from start (inclusive) to end (exclusive)

func (*Lexer) Messages

func (lex *Lexer) Messages() []errors.ErrorMessage

func (*Lexer) Next

func (lex *Lexer) Next() (tok token.Token, ok bool)

func (*Lexer) SetKeepComments

func (lex *Lexer) SetKeepComments(truthy bool)

func (*Lexer) SetSpaceWhitespace

func (lex *Lexer) SetSpaceWhitespace()

func (*Lexer) SetTabWhitespace

func (lex *Lexer) SetTabWhitespace()

func (*Lexer) String

func (lex *Lexer) String() string

converts receiver to string

intended for debugging and fail messages for tests

func (*Lexer) Tokenize

func (lex *Lexer) Tokenize() (tokens []token.Token, ok bool)

tokenize lex source

func (*Lexer) ValidLine

func (lex *Lexer) ValidLine() bool

returns true if and only if line number is a valid line position for lex's source

func (*Lexer) Write

func (lex *Lexer) Write() int

writes contents of input to lexer's source slice

returns number of lines written else -1 on error