syntax

README

syntax

General tokenizer and parser, written in Go

Documentation

Index

• type Lexer
  • func NewLexer(path string) (*Lexer, error)
  • func NewLexerFrom(text string) (*Lexer, error)
  • func (lexer *Lexer) Tokenize(text string, ignoreTypes []string) ([]Token, error)
• type Parser
  • func NewParser(path string) (*Parser, error)
  • func NewParserFrom(text string) (*Parser, error)
  • func (parser *Parser) CheckSyntax(text string, ignoreTypes []string) error
• type Sentence
• type Terminal
• type Token
  • func (t Token) Coords() string
  • func (t Token) String() string
• type Variable

Types

type Lexer

type Lexer struct {
	// contains filtered or unexported fields
}

func NewLexer

func NewLexer(path string) (*Lexer, error)

NewLexer creates a new Lexer with token types and patterns from the given file path

func NewLexerFrom

func NewLexerFrom(text string) (*Lexer, error)

NewLexerFrom creates a new Lexer with token types and patterns from the given text

func (*Lexer) Tokenize

func (lexer *Lexer) Tokenize(text string, ignoreTypes []string) ([]Token, error)

Tokenize tokenizes the given string and returns the list of Tokens. We can pass in a list of TokenTypes to ignore (pass nil if nothing to ignore).

type Parser

type Parser struct {
	// contains filtered or unexported fields
}

func NewParser

func NewParser(path string) (*Parser, error)

NewParser creates a new Parser from the given file path

func NewParserFrom

func NewParserFrom(text string) (*Parser, error)

NewParserFrom creates a new Parser from the given text

func (*Parser) CheckSyntax

func (parser *Parser) CheckSyntax(text string, ignoreTypes []string) error

type Sentence

type Sentence = []string

type Terminal

type Terminal = string

type Token

type Token struct {
	Type     string
	Text     string
	Row      int
	ColStart int
	ColEnd   int
}

func (Token) Coords

func (t Token) Coords() string

Coords returns the (Row: ColStart-ColEnd) as string

func (Token) String

func (t Token) String() string

String returns the string representation of Token

type Variable

type Variable = string