Documentation
¶
Overview ¶
Package token defines constants representing the lexical tokens of Delphi language and basic operations on tokens (printing, predicates).
Index ¶
- Constants
- type File
- func (f *File) AddLine(offset int)
- func (f *File) Base() int
- func (f *File) Line(p Pos) int
- func (f *File) LineCount() int
- func (f *File) MergeLine(line int)
- func (f *File) Name() string
- func (f *File) Offset(p Pos) int
- func (f *File) Pos(offset int) Pos
- func (f *File) Position(p Pos) (pos Position)
- func (f *File) SetLines(lines []int) bool
- func (f *File) SetLinesForContent(content []byte)
- func (f *File) Size() int
- type FileSet
- func (s *FileSet) AddFile(filename string, base, size int) *File
- func (s *FileSet) Base() int
- func (s *FileSet) File(p Pos) (f *File)
- func (s *FileSet) Iterate(f func(*File) bool)
- func (s *FileSet) Position(p Pos) (pos Position)
- func (s *FileSet) Read(decode func(interface{}) error) error
- func (s *FileSet) Write(encode func(interface{}) error) error
- type Pos
- type Position
- type Token
Constants ¶
const ( LowestPrec = 0 // non-operators UnaryPrec = 4 HighestPrec = 5 )
A set of constants for precedence-based expression parsing. Non-operators have lowest precedence, followed by operators starting with precedence 1 up to unary operators. The highest precedence serves as "catch-all" precedence for selector, indexing, and other operator and delimiter tokens.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type File ¶
type File struct {
// contains filtered or unexported fields
}
A File is a handle for a file belonging to a FileSet. A File has a name, size, and line offset table.
func (*File) AddLine ¶
AddLine adds the line offset for a new line. The line offset must be larger than the offset for the previous line and smaller than the file size; otherwise the line offset is ignored.
func (*File) Line ¶
Line returns the line number for the given file position p; p must be a Pos value in that file or NoPos.
func (*File) MergeLine ¶
MergeLine merges a line with the following line. It is akin to replacing the newline character at the end of the line with a space (to not change the remaining offsets). To obtain the line number, consult e.g. Position.Line. MergeLine will panic if given an invalid line number.
func (*File) Offset ¶
Offset returns the offset for the given file position p; p must be a valid Pos value in that file. f.Offset(f.Pos(offset)) == offset.
func (*File) Pos ¶
Pos returns the Pos value for the given file offset; the offset must be <= f.Size(). f.Pos(f.Offset(p)) == p.
func (*File) Position ¶
Position returns the Position value for the given file position p. p must be a Pos value in f or NoPos.
func (*File) SetLines ¶
SetLines sets the line offsets for a file and reports whether it succeeded. The line offsets are the offsets of the first character of each line; for instance for the content "ab\nc\n" the line offsets are {0, 3}. An empty file has an empty line offset table. Each line offset must be larger than the offset for the previous line and smaller than the file size; otherwise SetLines fails and returns false. Callers must not mutate the provided slice after SetLines returns.
func (*File) SetLinesForContent ¶
SetLinesForContent sets the line offsets for the given file content.
type FileSet ¶
type FileSet struct {
// contains filtered or unexported fields
}
A FileSet represents a set of source files. Methods of file sets are synchronized; multiple goroutines may invoke them concurrently.
func (*FileSet) AddFile ¶
AddFile adds a new file with a given filename, base offset, and file size to the file set s and returns the file. Multiple files may have the same name. The base offset must not be smaller than the FileSet's Base(), and size must not be negative. As a special case, if a negative base is provided, the current value of the FileSet's Base() is used instead.
Adding the file will set the file set's Base() value to base + size + 1 as the minimum base value for the next file. The following relationship exists between a Pos value p for a given file offset offs:
int(p) = base + offs
with offs in the range [0, size] and thus p in the range [base, base+size]. For convenience, File.Pos may be used to create file-specific position values from a file offset.
func (*FileSet) Base ¶
Base returns the minimum base offset that must be provided to AddFile when adding the next file.
func (*FileSet) File ¶
File returns the file that contains the position p. If no such file is found (for instance for p == NoPos), the result is nil.
func (*FileSet) Iterate ¶
Iterate calls f for the files in the file set in the order they were added until f returns false.
type Pos ¶
type Pos int
Pos is a compact encoding of a source position within a file set. It can be converted into a Position for a more convenient, but much larger, representation.
The Pos value for a given file is a number in the range [base, base+size], where base and size are specified when adding the file to the file set via AddFile.
To create the Pos value for a specific source offset (measured in bytes), first add the respective file to the current file set using FileSet.AddFile and then call File.Pos(offset) for that file. Given a Pos value p for a specific file set fset, the corresponding Position value is obtained by calling fset.Position(p).
Pos values can be compared directly with the usual comparison operators: If two Pos values p and q are in the same file, comparing p and q is equivalent to comparing the respective source file offsets. If p and q are in different files, p < q is true if the file implied by p was added to the respective file set before the file implied by q.
const NoPos Pos = 0
The zero value for Pos is NoPos; there is no file and line information associated with it, and NoPos().IsValid() is false. NoPos is always smaller than any other Pos value. The corresponding Position value for NoPos is the zero value for Position.
type Position ¶
type Position struct {
Filename string // filename, if any
Offset int // offset, starting at 0
Line int // line number, starting at 1
Column int // column number, starting at 1 (byte count)
}
Position describes an arbitrary source position including the file, line, and column location. A Position is valid if the line number is > 0.
type Token ¶
type Token int
Token is the set of lexical tokens of the Go programming language.
const ( // Special tokens ILLEGAL Token = iota EOF COMMENT // {} (* *) // CDIRECTIVE // {$I HELLO} {IFDEF X} // Identifiers and basic type literals // (these tokens stand for classes of literals) IDENT INTEGER // 12, $12A FLOAT // 0.412, -1e6 CHAR // #13, #$1A, ^C STRING // 'x', 'hello world' // Operators and delimiters AT // @ HAT // ^ ADD // + SUB // - MUL // * FDIV // / EQL // = LSS // < GTR // > NEQ // <> LEQ // <= GEQ // >= ASSIGN // := COLON // : COMMA // , PERIOD // . ELLIPSIS // .. SEMICOLON // ; LPAREN // ( LBRACK // [ RPAREN // ) RBRACK // ] // Keywords, excluding STRING AND ARRAY AS ASM BEGIN CASE CLASS CONST CONSTRUCTOR DESTRUCTOR DISPINTERFACE DIV DO DOWNTO ELSE END EXCEPT EXPORTS FILE FINALIZATION FINALLY FOR FUNCTION GOTO IF IMPLEMENTATION IN INHERITED INITIALIZATION INLINE INTERFACE IS LABEL LIBRARY MOD NIL NOT OBJECT OF OR PACKED PROCEDURE PROGRAM PROPERTY RAISE RECORD REPEAT RESOURCESTRING SET SHL SHR THEN THREADVAR TO TRY TYPE UNIT UNTIL USES VAR WHILE WITH XOR // Directives ABSOLUTE ABSTRACT ASSEMBLER AUTOMATED CDECL CONTAINS7 DEFAULT DELAYED11 DEPRECATED DISPID DYNAMIC EXPERIMENTAL EXPORT EXTERNAL FAR1 FINAL FORWARD HELPER8 IMPLEMENTS INDEX INLINE2 LIBRARY3 LOCAL4 MESSAGE NAME NEAR1 NODEFAULT OPERATOR10 OUT OVERLOAD OVERRIDE7 PACKAGE PASCAL PLATFORM PRIVATE PROTECTED PUBLIC PUBLISHED READ READONLY REFERENCE9 REGISTER REINTRODUCE REQUIRES7 RESIDENT1 SAFECALL SEALED5 STATIC STDCALL STORED STRICT UNSAFE VARARGS VIRTUAL WINAPI6 WRITE WRITEONLY )
The list of tokens.
func (Token) IsKeyword ¶
IsKeyword returns true for tokens corresponding to keywords; it returns false otherwise.
func (Token) IsLiteral ¶
IsLiteral returns true for tokens corresponding to identifiers and basic type literals; it returns false otherwise.
func (Token) IsOperator ¶
IsOperator returns true for tokens corresponding to operators and delimiters; it returns false otherwise.
func (Token) Precedence ¶
Precedence returns the operator precedence of the binary operator op. If op is not a binary operator, the result is LowestPrecedence.
func (Token) String ¶
String returns the string corresponding to the token tok. For operators, delimiters, and keywords the string is the actual token character sequence (e.g., for the token ADD, the string is "+"). For all other tokens the string corresponds to the token constant name (e.g. for the token IDENT, the string is "IDENT").
Source Files