scanner

Documentation

Overview

Package scanner provides a scanner and tokenizer for UTF-8-encoded text. It takes an io.Reader providing the source, which then can be tokenized through repeated calls to the Scan function. For compatibility with existing tools, the NUL character is not allowed. If the first character in the source is a UTF-8 encoded byte order mark (BOM), it is discarded.

Index

• func PrintError(w io.Writer, err error)
• type Error
  • func (e Error) Error() string
• type ErrorList
  • func (p *ErrorList) Add(pos token.Position, msg string)
  • func (p ErrorList) Err() error
  • func (p ErrorList) Error() string
  • func (p ErrorList) Len() int
  • func (p ErrorList) Less(i, j int) bool
  • func (p *ErrorList) RemoveMultiples()
  • func (p *ErrorList) Reset()
  • func (p ErrorList) Sort()
  • func (p ErrorList) Swap(i, j int)
• type Scanner
  • func (s *Scanner) Init(src io.Reader, filename string) *Scanner
  • func (s *Scanner) Peek() rune
  • func (s *Scanner) Pos() (pos token.Position)
  • func (s *Scanner) Scan() token.Token
  • func (s *Scanner) TokenText() string

Functions

func PrintError

func PrintError(w io.Writer, err error)

PrintError is a utility function that prints a list of errors to w, one error per line, if the err parameter is an ErrorList. Otherwise it prints the err string.

Types

type Error

type Error struct {
	Pos token.Position
	Msg string
}

In an ErrorList, an error is represented by an *Error. The position Pos, if valid, points to the beginning of the offending token, and the error condition is described by Msg.

func (Error) Error

func (e Error) Error() string

Error implements the error interface.

type ErrorList

type ErrorList []*Error

ErrorList is a list of *Errors. The zero value for an ErrorList is an empty ErrorList ready to use.

func (*ErrorList) Add

func (p *ErrorList) Add(pos token.Position, msg string)

Add adds an Error with given position and error message to an ErrorList.

func (ErrorList) Err

func (p ErrorList) Err() error

Err returns an error equivalent to this error list. If the list is empty, Err returns nil.

func (ErrorList) Error

func (p ErrorList) Error() string

An ErrorList implements the error interface.

func (ErrorList) Len

func (p ErrorList) Len() int

ErrorList implements the sort Interface.

func (ErrorList) Less

func (p ErrorList) Less(i, j int) bool

func (*ErrorList) RemoveMultiples

func (p *ErrorList) RemoveMultiples()

RemoveMultiples sorts an ErrorList and removes all but the first error per line.

func (*ErrorList) Reset

func (p *ErrorList) Reset()

Reset resets an ErrorList to no errors.

func (ErrorList) Sort

func (p ErrorList) Sort()

Sort sorts an ErrorList. *Error entries are sorted by position, other errors are sorted by error message, and before any *Error entry.

func (ErrorList) Swap

func (p ErrorList) Swap(i, j int)

type Scanner

type Scanner struct {

	// Error is called for each error encountered. If no Error
	// function is set, the error is reported to os.Stderr.
	Error func(s *Scanner, msg string)

	// ErrorCount is incremented by one for each error encountered.
	ErrorCount int

	// Start position of most recently scanned token; set by Scan.
	// Calling Init or Next invalidates the position (Line == 0).
	// The Filename field is always left untouched by the Scanner.
	// If an error is reported (via Error) and Position is invalid,
	// the scanner is not inside a token. Call Pos to obtain an error
	// position in that case.
	Position token.Position
	// contains filtered or unexported fields
}

A Scanner implements reading of Unicode characters and tokens from an io.Reader.

func (*Scanner) Init

func (s *Scanner) Init(src io.Reader, filename string) *Scanner

Init initializes a Scanner with a new source and returns s. Error is set to nil, ErrorCount is set to 0.

func (*Scanner) Peek

func (s *Scanner) Peek() rune

Peek returns the next Unicode character in the source without advancing the scanner. It returns -1 if the scanner's position is at the last character of the source.

func (*Scanner) Pos

func (s *Scanner) Pos() (pos token.Position)

Pos returns the position of the character immediately after the character or token returned by the last call to Next or Scan.

func (*Scanner) Scan

func (s *Scanner) Scan() token.Token

Scan reads the next token or Unicode character from source and returns it. It returns EOF at the end of the source. It reports scanner errors (read and token errors) by calling s.Error, if not nil; otherwise it prints an error message to os.Stderr.

func (*Scanner) TokenText

func (s *Scanner) TokenText() string

TokenText returns the string corresponding to the most recently scanned token. Valid after calling Scan().