token

README

token

Package token is a variant of the stdlib package token with types FileSet and Token removed

Installation

$ go get [-u] modernc.org/token

Documentation: godoc.org/modernc.org/token

Documentation

Overview

Package token is variant of the stdlib package token with types FileSet and Token removed.

Index

• type File
  • func NewFile(filename string, size int) *File
  • func (f *File) AddLine(offset int)
  • func (f *File) AddLineInfo(offset int, filename string, line int)
  • func (f *File) Base() int
  • func (f *File) Line(p Pos) int
  • func (f *File) LineCount() int
  • func (f *File) MergeLine(line int)
  • func (f *File) Name() string
  • func (f *File) Offset(p Pos) int
  • func (f *File) Pos(offset int) Pos
  • func (f *File) Position(p Pos) (pos Position)
  • func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)
  • func (f *File) SetLines(lines []int) bool
  • func (f *File) SetLinesForContent(content []byte)
  • func (f *File) Size() int
• type Pos
  • func (p Pos) IsValid() bool
• type Position
  • func (pos *Position) IsValid() bool
  • func (pos Position) String() string

Types

type File

type File struct {
	// contains filtered or unexported fields
}

A File is a handle for a file. A File has a name, size, and line offset table.

func NewFile

func NewFile(filename string, size int) *File

NewFile returns a new file with a given filename and file size.

func (*File) AddLine

func (f *File) AddLine(offset int)

AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the file size; otherwise the line offset is ignored.

func (*File) AddLineInfo

func (f *File) AddLineInfo(offset int, filename string, line int)

AddLineInfo adds alternative file and line number information for a given file offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the file size; otherwise the information is ignored.

AddLineInfo is typically used to register alternative position information for //line filename:line comments in source files.

func (*File) Base

func (f *File) Base() int

Base returns the base offset of file f as registered with AddFile.

func (*File) Line

func (f *File) Line(p Pos) int

Line returns the line number for the given file position p; p must be a Pos value in that file or NoPos.

func (*File) LineCount

func (f *File) LineCount() int

LineCount returns the number of lines in file f.

func (*File) MergeLine

func (f *File) MergeLine(line int)

MergeLine merges a line with the following line. It is akin to replacing the newline character at the end of the line with a space (to not change the remaining offsets). To obtain the line number, consult e.g. Position.Line. MergeLine will panic if given an invalid line number.

func (*File) Name

func (f *File) Name() string

Name returns the file name of file f as registered with AddFile.

func (*File) Offset

func (f *File) Offset(p Pos) int

Offset returns the offset for the given file position p; p must be a valid Pos value in that file. f.Offset(f.Pos(offset)) == offset.

func (*File) Pos

func (f *File) Pos(offset int) Pos

Pos returns the Pos value for the given file offset; the offset must be <= f.Size(). f.Pos(f.Offset(p)) == p.

func (*File) Position

func (f *File) Position(p Pos) (pos Position)

Position returns the Position value for the given file position p. Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).

func (*File) PositionFor

func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)

PositionFor returns the Position value for the given file position p. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a Pos value in f or NoPos.

func (*File) SetLines

func (f *File) SetLines(lines []int) bool

SetLines sets the line offsets for a file and reports whether it succeeded. The line offsets are the offsets of the first character of each line; for instance for the content "ab\nc\n" the line offsets are {0, 3}. An empty file has an empty line offset table. Each line offset must be larger than the offset for the previous line and smaller than the file size; otherwise SetLines fails and returns false. Callers must not mutate the provided slice after SetLines returns.

func (*File) SetLinesForContent

func (f *File) SetLinesForContent(content []byte)

SetLinesForContent sets the line offsets for the given file content. It ignores position-altering //line comments.

func (*File) Size

func (f *File) Size() int

Size returns the size of file f as registered with AddFile.

type Pos

type Pos int

Pos is a compact encoding of a source position within a file set. It can be converted into a Position for a more convenient, but much larger, representation.

The Pos value for a given file is a number in the range [base, base+size], where base and size are specified when adding the file to the file set via AddFile.

To create the Pos value for a specific source offset (measured in bytes), first add the respective file to the current file set using FileSet.AddFile and then call File.Pos(offset) for that file. Given a Pos value p for a specific file set fset, the corresponding Position value is obtained by calling fset.Position(p).

Pos values can be compared directly with the usual comparison operators: If two Pos values p and q are in the same file, comparing p and q is equivalent to comparing the respective source file offsets. If p and q are in different files, p < q is true if the file implied by p was added to the respective file set before the file implied by q.

const NoPos Pos = 0

NoPos is the zero value of Pos; there is no file and line information associated with it, and NoPos.IsValid() is false. NoPos is always smaller than any other Pos value. The corresponding Position value for NoPos is the zero value for Position.

func (Pos) IsValid

func (p Pos) IsValid() bool

IsValid reports whether the position is valid.

type Position

type Position struct {
	Filename string // filename, if any
	Offset   int    // offset, starting at 0
	Line     int    // line number, starting at 1
	Column   int    // column number, starting at 1 (byte count)
}

Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.

func (*Position) IsValid

func (pos *Position) IsValid() bool

IsValid reports whether the position is valid.

func (Position) String

func (pos Position) String() string

String returns a string in one of several forms:

file:line:column    valid position with file name
line:column         valid position without file name
file                invalid position with file name
-                   invalid position without file name