parser

Documentation

Overview

Package parser contains the framework for building a parser with the ptk package. The Parser, State, Table, and Entry types, along with the ExpectedToken, UnknownTokenType, and UnexpectedToken error generators are the most important types from this subpackage.

Index

• Variables
• func ExpectedToken(types ...string) error
• func UnexpectedToken(tok *common.Token, types ...string) error
• func UnknownTokenType(tok *common.Token, types ...string) error
• type Entry
• type ExprFirst
• type ExprNext
• type MockParser
  • func (m *MockParser) Expression(stream common.TokenStream) (common.Node, error)
  • func (m *MockParser) Statement(stream common.TokenStream) (common.Node, error)
  • func (m *MockParser) Statements(stream common.TokenStream) ([]common.Node, error)
• type MockState
  • func (m *MockState) Expression(p Parser, rbp int) (common.Node, error)
  • func (m *MockState) MoreTokens() bool
  • func (m *MockState) NextToken() *common.Token
  • func (m *MockState) PopStream() common.TokenStream
  • func (m *MockState) PopTable() Table
  • func (m *MockState) PushStream(ts common.TokenStream)
  • func (m *MockState) PushTable(tab Table)
  • func (m *MockState) PushToken(tok *common.Token)
  • func (m *MockState) SetStream(ts common.TokenStream) common.TokenStream
  • func (m *MockState) SetTable(tab Table) Table
  • func (m *MockState) Statement(p Parser) (common.Node, error)
  • func (m *MockState) Stream() common.TokenStream
  • func (m *MockState) Table() Table
  • func (m *MockState) Token() *common.Token
• type Parser
  • func New(table Table) Parser
• type State
  • func NewState(table Table, stream common.TokenStream) State
• type Statement
• type Table

Variables

var (
	ErrExpectedToken    = errors.New("No tokens available")
	ErrUnknownTokenType = errors.New("Unknown token type")
	ErrUnexpectedToken  = errors.New("Unexpected token")
	ErrNoTable          = errors.New("Programming error: Parse table missing")
)

Simple errors that may be generated within the package.

Functions

func ExpectedToken

func ExpectedToken(types ...string) error

ExpectedToken constructs and returns an ErrExpectedToken.

func UnexpectedToken

func UnexpectedToken(tok *common.Token, types ...string) error

UnexpectedToken cunstructs and returns an ErrUnexpectedToken.

func UnknownTokenType

func UnknownTokenType(tok *common.Token, types ...string) error

UnknownTokenType constructs and returns an ErrUnknownTokenType.

Types

type Entry

type Entry struct {
	Power int       // The binding power of the token type
	First ExprFirst // The function to call for an initial token
	Next  ExprNext  // The function to call for the next token
	Stmt  Statement // The function to call for statement tokens
}

Entry is an entry in the parser table. The Pratt technique is table driven, based on the token type; objects of this type contain a single entry from the table.

type ExprFirst

type ExprFirst func(p Parser, state State, power int, tok *common.Token) (common.Node, error)

ExprFirst functions are called to process the first token in an expression. Functions of this type are typically declared on literal tokens or prefix operators.

type ExprNext

type ExprNext func(p Parser, state State, power int, left common.Node, tok *common.Token) (common.Node, error)

ExprNext functions are called to process the subsequent tokens in an expression. Functions of this type are typically declared with a "left binding power" (a measure of how tightly an operator binds to its operands), and are used with binary operators.

type MockParser

type MockParser struct {
	mock.Mock
}

MockParser is a mock implementation of the Parser interface.

func (*MockParser) Expression

func (m *MockParser) Expression(stream common.TokenStream) (common.Node, error)

Expression parses a single expression from the specified token stream.

func (*MockParser) Statement

func (m *MockParser) Statement(stream common.TokenStream) (common.Node, error)

Statement parses a single statement from the specified token stream.

func (*MockParser) Statements

func (m *MockParser) Statements(stream common.TokenStream) ([]common.Node, error)

Statements parses all statements from the specified token stream. It is essentially equivalent to running Statement in a loop until all tokens are exhausted.

type MockState

type MockState struct {
	mock.Mock
}

MockState is a mock implementation of the State interface.

func (*MockState) Expression

func (m *MockState) Expression(p Parser, rbp int) (common.Node, error)

Expression parses a sub-expression. This is the core, workhorse function of expression parsing utilizing the Pratt technique; many token parsing functions end up calling this function recursively. It should be called with a "right binding power", which is used in operator precedence calculations.

func (*MockState) MoreTokens

func (m *MockState) MoreTokens() bool

MoreTokens returns a boolean true if there are more tokens available, that is, if NextToken will not return nil.

func (*MockState) NextToken

func (m *MockState) NextToken() *common.Token

NextToken returns the next token to be processed.

func (*MockState) PopStream

func (m *MockState) PopStream() common.TokenStream

PopStream allows popping a token stream off the token stream stack. Use this, paired with PushStream, when your grammar allows inclusion of alternate files. Note that PopStream is called implicitly if the token stream returns a nil.

func (*MockState) PopTable

func (m *MockState) PopTable() Table

PopTable allows popping a table pushed with PushTable off the parser table stack. Use this, paired with PushTable, when your grammar has different rules in particular sections of the file, e.g., like PHP embedded in a web page template.

func (*MockState) PushStream

func (m *MockState) PushStream(ts common.TokenStream)

PushStream allows pushing an alternative token stream onto the token stream stack. Use this, paired with PopStream, when your grammar allows inclusion of alternate files. Note that if the current token stream returns a nil, an implicit PopStream will be performed.

func (*MockState) PushTable

func (m *MockState) PushTable(tab Table)

PushTable allows pushing an alternative parser table onto the parser table stack. Use this, paired with PopTable, when your grammar has different rules in particular sections of the file, e.g., like PHP embedded in a web page template.

func (*MockState) PushToken

func (m *MockState) PushToken(tok *common.Token)

PushToken pushes a token back. This token will end up being the next token returned when NextToken is called. Note that the token returned by Token is not changed.

func (*MockState) SetStream

func (m *MockState) SetStream(ts common.TokenStream) common.TokenStream

SetStream allows changing the current token stream on the fly. Its action is similar to a PopStream followed by a PushStream, so the number of entries in the token stream stack remains the same.

func (*MockState) SetTable

func (m *MockState) SetTable(tab Table) Table

SetTable allows changing the current table on the fly. Its action is similar to a PopTable followed by a PushTable, so the number of entries in the parser table stack remains the same.

func (*MockState) Statement

func (m *MockState) Statement(p Parser) (common.Node, error)

Statement parses a single statement. This is the core, workhorse function for statement parsing utilizing the Pratt technique.

func (*MockState) Stream

func (m *MockState) Stream() common.TokenStream

Stream returns the token stream currently in use.

func (*MockState) Table

func (m *MockState) Table() Table

Table returns the parser table currently in use.

func (*MockState) Token

func (m *MockState) Token() *common.Token

Token returns the token currently being processed. It will be nil if NextToken has not been called, or if NextToken returned a nil value.

type Parser

type Parser interface {
	// Expression parses a single expression from the specified
	// token stream.
	Expression(stream common.TokenStream) (common.Node, error)

	// Statement parses a single statement from the specified
	// token stream.
	Statement(stream common.TokenStream) (common.Node, error)

	// Statements parses all statements from the specified token
	// stream.  It is essentially equivalent to running Statement
	// in a loop until all tokens are exhausted.
	Statements(stream common.TokenStream) ([]common.Node, error)
}

Parser represents the actual parser. This is passed to the parsing functions along with the parsing state.

func New

func New(table Table) Parser

New constructs a new parser, with the specified table.

type State

type State interface {
	// Table returns the parser table currently in use.
	Table() Table

	// PushTable allows pushing an alternative parser table onto
	// the parser table stack.  Use this, paired with PopTable,
	// when your grammar has different rules in particular
	// sections of the file, e.g., like PHP embedded in a web page
	// template.
	PushTable(tab Table)

	// PopTable allows popping a table pushed with PushTable off
	// the parser table stack.  Use this, paired with PushTable,
	// when your grammar has different rules in particular
	// sections of the file, e.g., like PHP embedded in a web page
	// template.
	PopTable() Table

	// SetTable allows changing the current table on the fly.  Its
	// action is similar to a PopTable followed by a PushTable, so
	// the number of entries in the parser table stack remains the
	// same.
	SetTable(tab Table) Table

	// Stream returns the token stream currently in use.
	Stream() common.TokenStream

	// PushStream allows pushing an alternative token stream onto
	// the token stream stack.  Use this, paired with PopStream,
	// when your grammar allows inclusion of alternate files.
	// Note that if the current token stream returns a nil, an
	// implicit PopStream will be performed.
	PushStream(ts common.TokenStream)

	// PopStream allows popping a token stream off the token
	// stream stack.  Use this, paired with PushStream, when your
	// grammar allows inclusion of alternate files.  Note that
	// PopStream is called implicitly if the token stream returns
	// a nil.
	PopStream() common.TokenStream

	// SetStream allows changing the current token stream on the
	// fly.  Its action is similar to a PopStream followed by a
	// PushStream, so the number of entries in the token stream
	// stack remains the same.
	SetStream(ts common.TokenStream) common.TokenStream

	// Token returns the token currently being processed.  It will
	// be nil if NextToken has not been called, or if NextToken
	// returned a nil value.
	Token() *common.Token

	// NextToken returns the next token to be processed.
	NextToken() *common.Token

	// MoreTokens returns a boolean true if there are more tokens
	// available, that is, if NextToken will not return nil.
	MoreTokens() bool

	// PushToken pushes a token back.  This token will end up
	// being the next token returned when NextToken is called.
	// Note that the token returned by Token is not changed.
	PushToken(tok *common.Token)

	// Expression parses a sub-expression.  This is the core,
	// workhorse function of expression parsing utilizing the
	// Pratt technique; many token parsing functions end up
	// calling this function recursively.  It should be called
	// with a "right binding power", which is used in operator
	// precedence calculations.
	Expression(p Parser, rbp int) (common.Node, error)

	// Statement parses a single statement.  This is the core,
	// workhorse function for statement parsing utilizing the
	// Pratt technique.
	Statement(p Parser) (common.Node, error)
}

State represents the parser state. This is passed to the parsing functions and should contain all the data they require to perform their operation.

func NewState

func NewState(table Table, stream common.TokenStream) State

NewState constructs and returns a new state, with the specified table and stream.

type Statement

type Statement func(p Parser, state State, tok *common.Token) (common.Node, error)

Statement functions are called to process a statement. They're called with the first token of the statement, and should read in additional tokens.

type Table

type Table map[string]Entry

Table is a table of entries by their token type. The Pratt technique is table driven, based on the token type; objects of this type contain the table.