scanner

Documentation

Overview

Package scanner provides the Scanner type that is able to read from any io.Reader and generate tokens.

Index

• type Scanner
  • func New(r io.Reader) *Scanner
  • func (s *Scanner) Scan() (tok Token)
• type Token
  • func (t Token) String() string

Types

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

A Scanner scans tokens from the io.Reader given at construction.

func New

func New(r io.Reader) *Scanner

New creates a new Scanner reading from the given io.Reader.

func (*Scanner) Scan

func (s *Scanner) Scan() (tok Token)

Scan returns the next token found in the given io.Reader. If an error occurs the Token will be of kind Illegal, and the text includes information about the error. If the io.Reader reaches EOF, the token will be of kind EOF.

type Token

type Token struct {
	token.Kind
	Text string
}

A Token is defined by its kind, and sometimes by some text.

func (Token) String

func (t Token) String() string

String returns a human readable representation of a Token.