Documentation
¶
Overview ¶
Package token defines constants representing the lexical tokens of the Go programming language and basic operations on tokens (printing, predicates).
Index ¶
- Constants
- Variables
- func WithinInclusive(offset int, start, end Pos) bool
- type File
- func (f *File) AddLine(offset int)
- func (f *File) AddLineInfo(offset int, filename string, line int)
- func (f *File) Base() intdeprecated
- func (f *File) Line(p Pos) int
- func (f *File) LineCount() int
- func (f *File) Lines() []int
- func (f *File) MergeLine(line int)
- func (f *File) Name() string
- func (f *File) Offset(p Pos) int
- func (f *File) Pos(offset int, rel RelPos) Pos
- func (f *File) Position(p Pos) (pos Position)
- func (f *File) PositionFor(p Pos, adjusted bool) (pos Position)
- func (f *File) SetLines(lines []int) bool
- func (f *File) SetLinesForContent(content []byte)
- func (f *File) Size() int
- type Pos
- func (p Pos) Add(n int) Pos
- func (p Pos) Before(q Pos) booldeprecated
- func (p Pos) Column() int
- func (p Pos) Compare(p2 Pos) int
- func (p Pos) File() *File
- func (p Pos) Filename() string
- func (p Pos) HasAbsPos() bool
- func (p Pos) HasRelPos() bool
- func (p Pos) IsNewline() bool
- func (p Pos) IsValid() bool
- func (p Pos) Line() int
- func (p Pos) Offset() int
- func (p Pos) Position() Position
- func (p Pos) RelPos() RelPos
- func (p Pos) String() string
- func (p Pos) WithRel(rel RelPos) Pos
- type Position
- type RelPos
- type Token
Constants ¶
const ( LowestPrec = lowestPrec UnaryPrec = unaryPrec HighestPrec = highestPrec )
A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.
Variables ¶
var NoPos = Pos{}
NoPos is the zero value for Pos; there is no file and line information associated with it, and Pos.IsValid is false.
NoPos is always larger than any valid Pos value, as it tends to relate to values produced from evaluating existing values with valid positions. The corresponding Position value for NoPos is the zero value.
Functions ¶
func WithinInclusive ¶ added in v0.16.0
WithinInclusive reports whether offset lies within the range start to end, inclusive on both ends. It is up to the caller to ensure that start and end are from the same file, and start is before end, and that offset is appropriate for the file.
Types ¶
type File ¶
type File struct {
// contains filtered or unexported fields
}
A File has a name, size, and line offset table.
func NewFile ¶
NewFile returns a new file with the given OS file name. The size provides the size of the whole file.
The second argument is deprecated. It has no effect.
func (*File) AddLine ¶
AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the file size; otherwise the line offset is ignored.
func (*File) AddLineInfo ¶
AddLineInfo adds alternative file and line number information for a given file offset. The offset must be larger than the offset for the previously added alternative line info and smaller than the file size; otherwise the information is ignored.
AddLineInfo is typically used to register alternative position information for //line filename:line comments in source files.
func (*File) Line ¶
Line returns the line number for the given file position p; p must be a Pos value in that file or NoPos.
func (*File) Lines ¶ added in v0.9.0
Lines returns the effective line offset table of the form described by File.SetLines. Callers must not mutate the result.
func (*File) MergeLine ¶
MergeLine merges a line with the following line. It is akin to replacing the newline character at the end of the line with a space (to not change the remaining offsets). To obtain the line number, consult e.g. Position.Line. MergeLine will panic if given an invalid line number.
func (*File) Offset ¶
Offset returns the offset for the given file position p.
If p is before the file's start position (or if p is NoPos), the result is 0; if p is past the file's end position, the the result is the file size (see also go.dev/issue/57490).
The following invariant, though not true for offset values in general, holds for the result offset: f.Offset(f.Pos(offset)) == offset
func (*File) Pos ¶
Pos returns the Pos value for the given file offset.
If offset is negative, the result is the file's start position; if the offset is too large, the result is the file's end position (see also go.dev/issue/57490).
The following invariant, though not true for Pos values in general, holds for the result p: f.Pos(f.Offset(p)) == p.
func (*File) Position ¶
Position returns the Position value for the given file position p. If p is out of bounds, it is adjusted to match the File.Offset behavior. Calling f.Position(p) is equivalent to calling f.PositionFor(p, true).
func (*File) PositionFor ¶
PositionFor returns the Position value for the given file position p. If p is out of bounds, it is adjusted to match the File.Offset behavior. If adjusted is set, the position may be adjusted by position-altering //line comments; otherwise those comments are ignored. p must be a Pos value in f or NoPos.
func (*File) SetLines ¶
SetLines sets the line offsets for a file and reports whether it succeeded. The line offsets are the offsets of the first character of each line; for instance for the content "ab\nc\n" the line offsets are {0, 3}. An empty file has an empty line offset table. Each line offset must be larger than the offset for the previous line and smaller than the file size; otherwise SetLines fails and returns false. Callers must not mutate the provided slice after SetLines returns.
func (*File) SetLinesForContent ¶
SetLinesForContent sets the line offsets for the given file content. It ignores position-altering //line comments.
type Pos ¶
type Pos struct {
// contains filtered or unexported fields
}
Pos is a compact encoding of a source position. When valid, as reported by Pos.IsValid, this can be either an absolute file position to obtain via Pos.Position, which can be rendered in a human-friendly text form, and/or a relative position to obtain via Pos.RelPos.
func (Pos) Before
deprecated
Before reports whether p < q, as documented in Pos.Compare.
Deprecated: use Pos.Compare instead.
func (Pos) Compare ¶ added in v0.12.0
Compare returns an integer comparing two positions. The result will be 0 if p == p2, -1 if p < p2, and +1 if p > p2. Note that NoPos is always larger than any valid position.
func (Pos) File ¶
File returns the file that contains the absolute position p or nil if there is no such file (for instance for p == NoPos).
func (Pos) HasAbsPos ¶ added in v0.16.0
HasAbsPos reports whether p has an absolute position, which can be obtained via Pos.Position.
func (Pos) HasRelPos ¶
HasRelPos reports whether p has a relative position, which can be obtained via Pos.RelPos.
func (Pos) IsNewline ¶
IsNewline reports whether the relative information suggests this node should be printed on a new line.
func (Pos) IsValid ¶
IsValid reports whether the position contains any useful information, meaning either an absolute file position (obtained via Pos.Position), and/or a relative position (obtained via Pos.RelPos).
type Position ¶
type Position struct {
Filename string // filename, if any
Offset int // offset, starting at 0
Line int // line number, starting at 1
Column int // column number, starting at 1 (byte count)
}
Position describes an arbitrary and absolute (printable) source position within a file, including offset, line, and column location, which can be rendered in a human-friendly text form.
A Position is valid if the line number is > 0.
type RelPos ¶
type RelPos int
RelPos indicates the relative position of token to the previous token.
const ( // NoRelPos indicates no relative position is specified. NoRelPos RelPos = iota // invalid // Elided indicates that the token for which this position is defined is // not rendered at all. Elided // elided // NoSpace indicates there is no whitespace before this token. NoSpace // nospace // Blank means there is horizontal space before this token. Blank // blank // Newline means there is a single newline before this token. Newline // newline // NewSection means there are two or more newlines before this token. NewSection // section )
type Token ¶
type Token int
Token is the set of lexical tokens of the CUE configuration language.
const ( // Special tokens ILLEGAL Token = iota EOF COMMENT // e.g. @foo(bar,baz=4) ATTRIBUTE // e.g. main, _tmp IDENT // e.g. 12_345Mi, 0700, 0xdeadbeef, 1.2M INT // e.g. 123.45 FLOAT // e.g. 3m4s; TODO // DURATION // e.g. "abc" STRING // a part of a template string, e.g. `"age: \(` INTERPOLATION BOTTOM // _|_ ADD // + SUB // - MUL // * POW // ^ QUO // / IQUO // quo IREM // rem IDIV // div IMOD // mod AND // & OR // | LAND // && LOR // || BIND // = EQL // == LSS // < GTR // > NOT // ! ARROW // <- NEQ // != LEQ // <= GEQ // >= MAT // =~ NMAT // !~ LPAREN // ( LBRACK // [ LBRACE // { COMMA // , PERIOD // . ELLIPSIS // ... RPAREN // ) RBRACK // ] RBRACE // } SEMICOLON // ; COLON // : OPTION // ? TILDE // ~ IF // if ELSE // else FOR // for IN // in LET // let TRY // try // TODO: remove in favor of OTHERWISE FALLBACK // fallback OTHERWISE // otherwise // experimental FUNC // func TRUE // true FALSE // false NULL // null )
The list of tokens.
func (Token) IsKeyword ¶
IsKeyword returns true for tokens corresponding to keywords; it returns false otherwise.
func (Token) IsLiteral ¶
IsLiteral returns true for tokens corresponding to identifiers and basic type literals; it returns false otherwise.
func (Token) IsOperator ¶
IsOperator returns true for tokens corresponding to operators and delimiters; it returns false otherwise.
func (Token) Precedence ¶
Precedence returns the operator precedence of the binary operator op. If op is not a binary operator, the result is LowestPrecedence.
Source Files