token

Documentation

Index

• type FileSet
  • func NewFileSet() *FileSet
  • func (s *FileSet) AddFile(filename string, base, size int) *SourceFile
  • func (s *FileSet) Base() int
  • func (s *FileSet) File(p Pos) (f *SourceFile)
  • func (s *FileSet) Iterate(f func(file *SourceFile) bool)
  • func (s *FileSet) Position(p Pos) (pos Position)
  • func (s *FileSet) PositionFor(p Pos, adjusted bool) (pos Position)
  • func (s *FileSet) RemoveFile(file *SourceFile)
• type Kind
  • func Lookup(ident string) Kind
  • func (k Kind) String() string
• type Pos
  • func (p Pos) IsValid() bool
• type Position
  • func (pos *Position) IsValid() bool
  • func (pos Position) String() string
• type SourceFile
  • func (f *SourceFile) AddLine(offset int)
  • func (f *SourceFile) AddLineColumnInfo(offset int, filename string, line, column int)
  • func (f *SourceFile) AddLineInfo(offset int, filename string, line int)
  • func (f *SourceFile) Base() int
  • func (f *SourceFile) Line(p Pos) int
  • func (f *SourceFile) LineCount() int
  • func (f *SourceFile) LineStart(line int) Pos
  • func (f *SourceFile) MergeLine(line int)
  • func (f *SourceFile) Name() string
  • func (f *SourceFile) Offset(p Pos) int
  • func (f *SourceFile) Pos(offset int) Pos
  • func (f *SourceFile) Position(p Pos) (pos Position)
  • func (f *SourceFile) PositionFor(p Pos, adjusted bool) (pos Position)
  • func (f *SourceFile) SetLines(lines []int) bool
  • func (f *SourceFile) SetLinesForContent(content []byte)
  • func (f *SourceFile) Size() int
• type Token
  • func NewToken(kind Kind, text string, pos *Position) Token
  • func (t *Token) String() string

Types

type FileSet

type FileSet struct {
	// contains filtered or unexported fields
}

A FileSet represents a set of source files. Methods of file sets are synchronized; multiple goroutines may invoke them concurrently.

The byte offsets for each file in a file set are mapped into distinct (integer) intervals, one interval [base, base+size] per file. Base represents the first byte in the file, and size is the corresponding file size. A Pos value is a value in such an interval. By determining the interval a Pos value belongs to, the file, its file base, and thus the byte offset (position) the Pos value is representing can be computed.

When adding a new file, a file base must be provided. That can be any integer value that is past the end of any interval of any file already in the file set. For convenience, FileSet.Base provides such a value, which is simply the end of the Pos interval of the most recently added file, plus one. Unless there is a need to extend an interval later, using the FileSet.Base should be used as argument for FileSet.AddFile.

A File may be removed from a FileSet when it is no longer needed. This may reduce memory usage in a long-running application.

func NewFileSet

func NewFileSet() *FileSet

NewFileSet creates a new file set.

func (*FileSet) AddFile

func (s *FileSet) AddFile(filename string, base, size int) *SourceFile

AddFile adds a new file with a given filename, base offset, and file size to the file set s and returns the file. Multiple files may have the same name. The base offset must not be smaller than the FileSet's Base(), and size must not be negative. As a special case, if a negative base is provided, the current value of the FileSet's Base() is used instead.

Adding the file will set the file set's Base() value to base + size + 1 as the minimum base value for the next file. The following relationship exists between a Pos value p for a given file offset offs:

int(p) = base + offs

with offs in the range [0, size] and thus p in the range [base, base+size]. For convenience, File.Pos may be used to create file-specific position values from a file offset.

func (*FileSet) Base

func (s *FileSet) Base() int

Base returns the minimum base offset that must be provided to AddFile when adding the next file.

func (*FileSet) File

func (s *FileSet) File(p Pos) (f *SourceFile)

File returns the file that contains the position p. If no such file is found (for instance for p == NoPos), the result is nil.

func (*FileSet) Iterate

func (s *FileSet) Iterate(f func(file *SourceFile) bool)

Iterate calls f for the files in the file set in the order they were added until f returns false.

func (*FileSet) Position

func (s *FileSet) Position(p Pos) (pos Position)

Position converts a Pos p in the fileset into a Position value. Calling s.Position(p) is equivalent to calling s.PositionFor(p, true).

func (*FileSet) PositionFor

func (s *FileSet) PositionFor(p Pos, adjusted bool) (pos Position)

PositionFor converts a Pos p in the fileset into a Position value. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a Pos value in s or NoPos.

func (*FileSet) RemoveFile

func (s *FileSet) RemoveFile(file *SourceFile)

RemoveFile removes a file from the FileSet so that subsequent queries for its Pos interval yield a negative result. This reduces the memory usage of a long-lived FileSet that encounters an unbounded stream of files.

Removing a file that does not belong to the set has no effect.

type Kind

type Kind byte

Kind defines the category of a token. E.g. Keyword, Identifier etc

const (
	Illegal Kind = iota
	EOF

	// Keywords
	Program
	Begin
	End
	Var
	Integer
	Goto
	Div
	Mod
	And
	Or
	In
	Function
	Procedure
	If
	Then
	Else
	With
	For
	Repeat
	While
	Case
	Nil
	Not
	To
	DownTo
	Do
	Const
	Until
	Of
	Type
	Array
	Record
	File
	Set
	Packed
	Real
	Boolean
	Char
	Label
	True
	False

	Identifier
	SemiColon
	StrLiteral
	Period
	LParen
	RParen
	Initialize
	Plus
	Colon
	UIntLiteral
	Comma
	Star
	FwdSlash
	Minus
	Equal
	LessThan
	NotEqual
	LessThanOrEqual
	GreaterThan
	GreaterThanOrEqual
	URealLiteral
	Range
	LSqBrace
	RSqBrace
	Caret
	Return
	Input
	Output
)

func Lookup

func Lookup(ident string) Kind

Lookup maps an identifier to its keyword token or IDENT (if not a keyword).

func (Kind) String

func (k Kind) String() string

type Pos

type Pos int

Pos is a compact encoding of a source position within a file set. It can be converted into a Position for a more convenient, but much larger, representation.

The Pos value for a given file is a number in the range [base, base+size], where base and size are specified when a file is added to the file set. The difference between a Pos value and the corresponding file base corresponds to the byte offset of that position (represented by the Pos value) from the beginning of the file. Thus, the file base offset is the Pos value representing the first byte in the file.

To create the Pos value for a specific source offset (measured in bytes), first add the respective file to the current file set using FileSet.AddFile and then call File.Pos(offset) for that file. Given a Pos value p for a specific file set fset, the corresponding Position value is obtained by calling fset.Position(p).

Pos values can be compared directly with the usual comparison operators: If two Pos values p and q are in the same file, comparing p and q is equivalent to comparing the respective source file offsets. If p and q are in different files, p < q is true if the file implied by p was added to the respective file set before the file implied by q.

const NoPos Pos = 0

NoPos

The zero value for Pos is NoPos; there is no file and line information associated with it, and NoPos.IsValid() is false. NoPos is always smaller than any other Pos value. The corresponding Position value for NoPos is the zero value for Position.

func (Pos) IsValid

func (p Pos) IsValid() bool

IsValid reports whether the position is valid.

type Position

type Position struct {
	Filename string // filename, if any
	Offset   int    // offset, starting at 0
	Line     int    // line number, starting at 1
	Column   int    // column number, starting at 1 (byte count)
}

Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.

func (*Position) IsValid

func (pos *Position) IsValid() bool

IsValid reports whether the position is valid.

func (Position) String

func (pos Position) String() string

String returns a string in one of several forms:

file:line:column    valid position with file name
file:line           valid position with file name but no column (column == 0)
line:column         valid position without file name
line                valid position without file name and no column (column == 0)
file                invalid position with file name
-                   invalid position without file name

type SourceFile

type SourceFile struct {
	// contains filtered or unexported fields
}

A SourceFile is a handle for a file belonging to a FileSet. A SourceFile has a name, size, and line offset table.

func (*SourceFile) AddLine

func (f *SourceFile) AddLine(offset int)

AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the file size; otherwise the line offset is ignored.

func (*SourceFile) AddLineColumnInfo

func (f *SourceFile) AddLineColumnInfo(offset int, filename string, line, column int)

AddLineColumnInfo adds alternative file, line, and column number information for a given file offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the file size; otherwise the information is ignored.

AddLineColumnInfo is typically used to register alternative position information for line directives such as //line filename:line:column.

func (*SourceFile) AddLineInfo

func (f *SourceFile) AddLineInfo(offset int, filename string, line int)

AddLineInfo is like AddLineColumnInfo with a column = 1 argument. It is here for backward-compatibility for code prior to Go 1.11.

func (*SourceFile) Base

func (f *SourceFile) Base() int

Base returns the base offset of file f as registered with AddFile.

func (*SourceFile) Line

func (f *SourceFile) Line(p Pos) int

Line returns the line number for the given file position p; p must be a Pos value in that file or NoPos.

func (*SourceFile) LineCount

func (f *SourceFile) LineCount() int

LineCount returns the number of lines in file f.

func (*SourceFile) LineStart

func (f *SourceFile) LineStart(line int) Pos

LineStart returns the Pos value of the start of the specified line. It ignores any alternative positions set using AddLineColumnInfo. LineStart panics if the 1-based line number is invalid.

func (*SourceFile) MergeLine

func (f *SourceFile) MergeLine(line int)

MergeLine merges a line with the following line. It is akin to replacing the newline character at the end of the line with a space (to not change the remaining offsets). To obtain the line number, consult e.g. Position.Line. MergeLine will panic if given an invalid line number.

func (*SourceFile) Name

func (f *SourceFile) Name() string

Name returns the file name of file f as registered with AddFile.

func (*SourceFile) Offset

func (f *SourceFile) Offset(p Pos) int

Offset returns the offset for the given file position p; p must be a valid Pos value in that file. f.Offset(f.Pos(offset)) == offset.

func (*SourceFile) Pos

func (f *SourceFile) Pos(offset int) Pos

Pos returns the Pos value for the given file offset; the offset must be <= f.Size(). f.Pos(f.Offset(p)) == p.

func (*SourceFile) Position

func (f *SourceFile) Position(p Pos) (pos Position)

Position returns the Position value for the given file position p. Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).

func (*SourceFile) PositionFor

func (f *SourceFile) PositionFor(p Pos, adjusted bool) (pos Position)

PositionFor returns the Position value for the given file position p. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a Pos value in f or NoPos.

func (*SourceFile) SetLines

func (f *SourceFile) SetLines(lines []int) bool

SetLines sets the line offsets for a file and reports whether it succeeded. The line offsets are the offsets of the first character of each line; for instance for the content "ab\nc\n" the line offsets are {0, 3}. An empty file has an empty line offset table. Each line offset must be larger than the offset for the previous line and smaller than the file size; otherwise SetLines fails and returns false. Callers must not mutate the provided slice after SetLines returns.

func (*SourceFile) SetLinesForContent

func (f *SourceFile) SetLinesForContent(content []byte)

SetLinesForContent sets the line offsets for the given file content. It ignores position-altering //line comments.

func (*SourceFile) Size

func (f *SourceFile) Size() int

Size returns the size of file f as registered with AddFile.

type Token

type Token struct {
	Text string
	Kind Kind
	Pos  *Position
}

Token defines a type of token

func NewToken

func NewToken(kind Kind, text string, pos *Position) Token

NewToken constructs and returns a new token

func (*Token) String

func (t *Token) String() string