scanner

Documentation

Overview

Package scanner tokenizes Writ source for the parser and for editors.

Index

• func IsBuiltin(name string) bool
• func IsCoreBuiltin(name string) bool
• func IsIntLit(word string) bool
• func IsKeyword(name string) bool
• func IsNumLit(word string) bool
• func IsSlot(word string) bool
• func Words() []string
• type Token
  • func Scan(src string) []Token
  • func Tokenize(src string) []Token
• type TokenKind
  • func (k TokenKind) String() string

Functions

func IsBuiltin

func IsBuiltin(name string) bool

IsBuiltin reports whether name is a core builtin (including true/false/nil).

func IsCoreBuiltin

func IsCoreBuiltin(name string) bool

IsCoreBuiltin reports whether name is a callable core builtin. true, false, and nil are builtins but not callable.

func IsIntLit

func IsIntLit(word string) bool

IsIntLit reports whether word is a signed integer literal.

func IsKeyword

func IsKeyword(name string) bool

IsKeyword reports whether name is a special form.

func IsNumLit

func IsNumLit(word string) bool

IsNumLit reports whether word is an integer or decimal float literal.

func IsSlot

func IsSlot(word string) bool

IsSlot reports whether word is a short-fn slot (#1, #2, …).

func Words

func Words() []string

Words returns keywords and builtins, sorted, for completion.

Types

type Token

type Token struct {
	Kind  TokenKind
	Text  string
	Start int
	End   int
}

Token is a source token. Text is the exact source slice.

func Scan

func Scan(src string) []Token

Scan splits src into tokens with positions. It does not fail on unterminated strings or ticks; the parser reports those.

func Tokenize

func Tokenize(src string) []Token

Tokenize splits src into highlighting tokens.

type TokenKind

type TokenKind int

TokenKind classifies a source token.

const (
	TokWS TokenKind = iota
	TokComment
	TokParen
	TokString
	TokNumber
	TokKeyword
	TokBuiltin
	TokSymbol
	TokLParen
	TokRParen
	TokLBracket
	TokRBracket
	TokQuote
	TokUnquote
	TokSplice
	TokTick
	TokEOF
)

func (TokenKind) String

func (k TokenKind) String() string