token

Documentation

Overview

Package token defines the token types for SQL parsing.

ANSI core tokens are defined as constants (IDs 0-999) for switch performance. Dialect-specific tokens are registered dynamically via Register().

Index

• func IsDynamic(t TokenType) bool
• func IsKeyword(t TokenType) bool
• func IsOperator(t TokenType) bool
• func RegisteredTokens() map[TokenType]string
• type Comment
  • func (c *Comment) IsBlockComment() bool
  • func (c *Comment) IsLineComment() bool
• type CommentKind
• type Position
  • func (p Position) IsValid() bool
• type Span
  • func (s Span) Contains(offset int) bool
  • func (s Span) IsValid() bool
• type Token
• type TokenType
  • func LookupDynamicKeyword(name string) (TokenType, bool)
  • func LookupIdent(ident string) TokenType
  • func Register(name string) TokenType
  • func (t TokenType) String() string

Functions

func IsDynamic

func IsDynamic(t TokenType) bool

IsDynamic returns true if the token type is a dynamically registered token.

func IsKeyword

func IsKeyword(t TokenType) bool

IsKeyword returns true if the token type is a keyword.

func IsOperator

func IsOperator(t TokenType) bool

IsOperator returns true if the token type is an operator.

func RegisteredTokens

func RegisteredTokens() map[TokenType]string

RegisteredTokens returns a copy of all registered dynamic tokens.

Types

type Comment

type Comment struct {
	Kind CommentKind
	Text string // includes delimiters (-- or /* */)
	Span Span
}

Comment represents a SQL comment with position.

func (*Comment) IsBlockComment

func (c *Comment) IsBlockComment() bool

IsBlockComment returns true if this is a block comment.

func (*Comment) IsLineComment

func (c *Comment) IsLineComment() bool

IsLineComment returns true if this is a line comment.

type CommentKind

type CommentKind int

CommentKind distinguishes line vs block comments.

const (
	LineComment  CommentKind = iota // -- comment
	BlockComment                    // /* comment */
)

Comment kinds.

type Position

type Position struct {
	Line   int // 1-based line number
	Column int // 1-based column number
	Offset int // 0-based byte offset
}

Position represents a location in the source code.

func (Position) IsValid

func (p Position) IsValid() bool

IsValid returns true if the position is valid (line > 0).

type Span

type Span struct {
	Start Position
	End   Position
}

Span represents a range in source code.

func (Span) Contains

func (s Span) Contains(offset int) bool

Contains returns true if the span contains the given offset.

func (Span) IsValid

func (s Span) IsValid() bool

IsValid returns true if both start and end positions are valid.

type Token

type Token struct {
	Type    TokenType
	Literal string
	Pos     Position
}

Token represents a lexical token with position information.

type TokenType

type TokenType int32

TokenType represents the type of a lexical token. Note: Named TokenType instead of Type because it's used extensively across the codebase and changing it would require a large refactor.

const (
	// Special tokens
	EOF TokenType = iota
	ILLEGAL

	// Literals
	IDENT  // identifier
	NUMBER // 123, 45.67, 1e10
	STRING // 'hello'

	// Operators (ANSI)
	PLUS     // +
	MINUS    // -
	STAR     // *
	SLASH    // /
	MOD      // %
	DPIPE    // ||
	EQ       // =
	NE       // != or <>
	LT       // <
	GT       // >
	LE       // <=
	GE       // >=
	DOT      // .
	COMMA    // ,
	LPAREN   // (
	RPAREN   // )
	LBRACKET // [
	RBRACKET // ]
	LBRACE   // {
	RBRACE   // }
	COLON    // :
	ARROW    // ->

	// ANSI Keywords (alphabetical)
	ALL
	AND
	AS
	ASC
	BETWEEN
	BY
	CASE
	CAST
	CROSS
	CURRENT
	DESC
	DISTINCT
	ELSE
	END
	EXISTS
	EXCEPT
	FALSE
	FETCH
	FILTER
	FIRST
	FOLLOWING
	FROM
	FULL
	GROUP
	GROUPS
	HAVING
	IN
	INNER
	INTERSECT
	IS
	JOIN
	LAST
	LATERAL
	LEFT
	LIKE
	LIMIT
	NATURAL
	NEXT
	NOT
	NULL
	NULLS
	OFFSET
	ON
	ONLY
	OR
	ORDER
	OUTER
	OVER
	PARTITION
	PERCENT
	PRECEDING
	RANGE
	RECURSIVE
	RIGHT
	ROW
	ROWS
	SELECT
	THEN
	TIES
	TRUE
	UNBOUNDED
	UNION
	USING
	WHEN
	WHERE
	WINDOW // Named window definitions
	WITH
	WITHIN

	// Extended SQL tokens (not ANSI-92, but common in modern dialects)
	QUALIFY   // DuckDB, Databricks, Snowflake, BigQuery
	ILIKE     // DuckDB, Postgres, Databricks, Snowflake
	RETURNING // Postgres, SQLite, DuckDB
	SEMI      // SEMI JOIN (DuckDB, Databricks, Spark)
	ANTI      // ANTI JOIN (DuckDB, Databricks, Spark)

	// Extended operators
	DCOLON // :: cast operator (Postgres, DuckDB, Databricks)

	// Template tokens
	MACRO // {{ ... }} content

)

func LookupDynamicKeyword

func LookupDynamicKeyword(name string) (TokenType, bool)

LookupDynamicKeyword returns the token type for a dynamic keyword. The lookup is case-insensitive. Returns IDENT and false if the keyword is not registered.

func LookupIdent

func LookupIdent(ident string) TokenType

LookupIdent returns the token type for the given identifier. If the identifier is a keyword, the keyword token type is returned. Otherwise, IDENT is returned. This only checks builtin keywords; use LookupIdentWithDialect for dynamic keywords.

func Register

func Register(name string) TokenType

Register registers a dynamic token with the given name. If a token with the same name already exists, returns the existing ID. This allows multiple dialects to register the same keyword (e.g., ILIKE) and share the same token ID.

The name is normalized to lowercase for lookup but stored in uppercase for display. Thread-safe: Uses mutex to protect concurrent registration.

func (TokenType) String

func (t TokenType) String() string

String returns a human-readable representation of the token type.