tokenizer

Documentation

Overview

Package tokenizer contains the tokenizer we use for parsing BASIC programs.

Given a string containing a complete BASIC program this package allows that to be iterated over as a series of tokens.

Our interpeter is intentionally naive, and executes tokens directly, without any intermediary representation.

Index

• type Tokenizer
  • func New(input string) *Tokenizer
  • func (l *Tokenizer) NextToken() token.Token

Types

type Tokenizer

type Tokenizer struct {
	// contains filtered or unexported fields
}

Tokenizer holds our state.

func New

func New(input string) *Tokenizer

New returns a Tokenizer instance from the specified string input.

func (*Tokenizer) NextToken

func (l *Tokenizer) NextToken() token.Token

NextToken reads and returns the next available token, skipping any white space which might be present.