scanner

README

scanner

import path "modernc.org/scanner"

Package scanner provides some common scanner stuff.

Documentation

Overview

Package scanner provides some common scanner stuff.

Index

• type ErrList
  • func (e *ErrList) AddErr(pos token.Position, msg string, args ...interface{})
  • func (e ErrList) Err() (r error)
  • func (e ErrList) Error() string
• type ErrWithPosition
  • func (e ErrWithPosition) Error() string
• type RecScanner
  • func NewRecScanner(name string, buf []byte, scan func(s []byte) (id, length int), whiteSpace int) *RecScanner
  • func (s *RecScanner) AddErr(pos token.Position, msg string, args ...interface{})
  • func (s *RecScanner) AddLine(offset int)
  • func (s *RecScanner) AddLineColumnInfo(offset int, filename string, line, column int)
  • func (s *RecScanner) Ch(n int) rune
  • func (s *RecScanner) Err() error
  • func (s *RecScanner) Len() int
  • func (s *RecScanner) Position(offset int) (r token.Position)
  • func (s *RecScanner) Scan() Token
  • func (s *RecScanner) Token(n int) Token
• type ScanSep
• type ScanSrc
• type Scanner
  • func NewScanner(name string, buf []byte, scanSep ScanSep, scanSrc ScanSrc) *Scanner
  • func (s *Scanner) AddErr(pos token.Position, msg string, args ...interface{})
  • func (s *Scanner) AddLine(offset int)
  • func (s *Scanner) AddLineColumnInfo(offset int, filename string, line, column int)
  • func (s *Scanner) Ch(n int) rune
  • func (s *Scanner) Err() error
  • func (s *Scanner) Len() int
  • func (s *Scanner) Position(offset int) (r token.Position)
  • func (s *Scanner) Scan() Token
  • func (s *Scanner) Token(n int) Token
• type Token
  • func (t Token) IsValid() bool
  • func (t Token) Next() (r Token)
  • func (t Token) Position() (r token.Position)
  • func (t Token) Prev() (r Token)
  • func (t Token) Sep() string
  • func (t Token) SetSep(s string)
  • func (t Token) SetSrc(s string)
  • func (t Token) Src() string
  • func (t Token) String() string

Types

type ErrList

type ErrList []ErrWithPosition

ErrList is a list of errors.

func (*ErrList) AddErr

func (e *ErrList) AddErr(pos token.Position, msg string, args ...interface{})

AddErr adds an error message associated with an optional position.

func (ErrList) Err

func (e ErrList) Err() (r error)

Err returns e if len(e) != 0 or nil.

func (ErrList) Error

func (e ErrList) Error() string

Error implements error.

type ErrWithPosition

type ErrWithPosition struct {
	Pos token.Position
	Err error
}

ErrWithPosition augments an error with optional position information.

func (ErrWithPosition) Error

func (e ErrWithPosition) Error() string

Error implements error.

type RecScanner

type RecScanner struct {
	// contains filtered or unexported fields
}

RecScanner represents the data structures and methods common to some/many lexical scanners, specialized for using scan functions produced by the modernc.org/rec compiler.

func NewRecScanner

func NewRecScanner(name string, buf []byte, scan func(s []byte) (id, length int), whiteSpace int) *RecScanner

NewRecScanner returns a newly created RecScanner. The 'name' argument is used to report positions. 'buf' becomes owned by the scanner and should not be mutated by the caller afterwards.

The 'scan' function that is compatible with functions that the modernc.org/rec compiler produces. 'whiteSpace' is the id the 'scan' function returns for white space. The production for white space does not have to handle sequences of white space. RecScanner handles sequences of white space automatically. You can still write your regular expression for white_space like for example

`(// |\t|\n|\r)*`

or

`( |\t|\n|\r)*`

But you can also write it simply as

` |\t|\n|\r`

with the same effect. This helps avoiding the problems described at egg issue 1.

Calling AddLine is done automatically by the RecScanner.Scan method.

func (*RecScanner) AddErr

func (s *RecScanner) AddErr(pos token.Position, msg string, args ...interface{})

AddErr registers an error.

func (*RecScanner) AddLine

func (s *RecScanner) AddLine(offset int)

AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the scanner buffer size; otherwise the line offset is ignored.

func (*RecScanner) AddLineColumnInfo

func (s *RecScanner) AddLineColumnInfo(offset int, filename string, line, column int)

AddLineColumnInfo adds alternative file, line, and column number information for a given scanner buffer offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the scanner buffer size; otherwise the information is ignored.

AddLineColumnInfo is typically used to register alternative position information for line directives such as //line filename:line:column.

func (*RecScanner) Ch

func (s *RecScanner) Ch(n int) rune

Ch returns the Ch field of the n-th token in 's'. Ch panics if n is out of range [0..Len()-1].

func (*RecScanner) Err

func (s *RecScanner) Err() error

Err reports any errors from reported by AddErr()

func (*RecScanner) Len

func (s *RecScanner) Len() int

Len reports the number of tokens in 's'.

func (*RecScanner) Position

func (s *RecScanner) Position(offset int) (r token.Position)

Position returns the position determined by offset.

func (*RecScanner) Scan

func (s *RecScanner) Scan() Token

Scan returns the next token.

func (*RecScanner) Token

func (s *RecScanner) Token(n int) Token

Token returns the n-th token in 's'. Ch panics if n is out of range [0..Len()-1].

type ScanSep

type ScanSep func() int

ScanSep is a function used by Scanner to determine the length in bytes of any separator preceding a token or EOF.

If the buf contains "if foo()" and the current offset is 2, calling

ScanSep()

may return

1

to indicate the separator is " ".

Scanner increments its own kept buffer offset by the returned length, the ScanSep function is assumed to do the same.

type ScanSrc

type ScanSrc func() (int, rune)

ScanSrc is a function used by Scanner to determine the length in bytes of a token.

If the buf contains "if foo()" and the current offset is 3, calling

ScanSrc()

may return

(3, IDENT)

to indicate the token is "foo" and its semantic value is IDENT.

Once ScanSrc returns zero for the token length an EOF is assumed.

Scanner increments its own kept buffer offset by the returned length, the ScanSrc function is assumed to do the same.

type Scanner

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner represents the data structures and methods common to some/many lexical scanners.

func NewScanner

func NewScanner(name string, buf []byte, scanSep ScanSep, scanSrc ScanSrc) *Scanner

NewScanner returns a newly created Scanner. The 'name' argument is used to report positions. 'buf' becomes owned by the scanner and should not be mutated by the caller afterwards.

func (*Scanner) AddErr

func (s *Scanner) AddErr(pos token.Position, msg string, args ...interface{})

AddErr registers an error.

func (*Scanner) AddLine

func (s *Scanner) AddLine(offset int)

AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the scanner buffer size; otherwise the line offset is ignored.

func (*Scanner) AddLineColumnInfo

func (s *Scanner) AddLineColumnInfo(offset int, filename string, line, column int)

AddLineColumnInfo adds alternative file, line, and column number information for a given scanner buffer offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the scanner buffer size; otherwise the information is ignored.

AddLineColumnInfo is typically used to register alternative position information for line directives such as //line filename:line:column.

func (*Scanner) Ch

func (s *Scanner) Ch(n int) rune

Ch returns the Ch field of the n-th token in 's'. Ch panics if n is out of range [0..Len()-1].

func (*Scanner) Err

func (s *Scanner) Err() error

Err reports any errors from reported by AddErr()

func (*Scanner) Len

func (s *Scanner) Len() int

Len reports the number of tokens in 's'.

func (*Scanner) Position

func (s *Scanner) Position(offset int) (r token.Position)

Position returns the position determined by offset.

func (*Scanner) Scan

func (s *Scanner) Scan() Token

Scan returns the next token.

func (*Scanner) Token

func (s *Scanner) Token(n int) Token

Token returns the n-th token in 's'. Ch panics if n is out of range [0..Len()-1].

type Token

type Token struct {
	// Ch represents the semantic value of the token as determined by the Scan
	// function.
	Ch rune
	// contains filtered or unexported fields
}

Token represents a lexeme, its position and semantic value.

func (Token) IsValid

func (t Token) IsValid() bool

IsValid reports whether t is a valid token. Zero values of Token report false.

func (Token) Next

func (t Token) Next() (r Token)

Next returns the token following t or a zero value if no such token exists.

func (Token) Position

func (t Token) Position() (r token.Position)

Position reports the position of t.

func (Token) Prev

func (t Token) Prev() (r Token)

Prev returns the token preceding t or a zero value if no such token exists.

func (Token) Sep

func (t Token) Sep() string

Sep returns the separator preceding t.

func (Token) SetSep

func (t Token) SetSep(s string)

SetSep sets t's separator.

func (Token) SetSrc

func (t Token) SetSrc(s string)

SetSrc sets t's source form.

func (Token) Src

func (t Token) Src() string

Src returns t's source form, without its preceding separator.

func (Token) String

func (t Token) String() string

String implements fmt.Stringer.