token

Documentation

Overview

Package token declares the token kind constants shared by the lexer and tokenize packages.

Mirrors CPython's split between the C tokenizer's enum (in Include/internal/pycore_token.h) and the Python-level Lib/token.py re-export. Keeping the kinds in their own package lets the lexer produce them and the tokenize wrapper consume them without an import cycle.

CPython: Include/internal/pycore_token.h Token enum CPython: Lib/token.py

Index

• type Type
  • func (t Type) String() string

Types

type Type

type Type int

Type is the token kind. Numeric values match CPython 3.14 Grammar/Tokens one for one. The full constant set is in types_gen.go.

const (
	ENDMARKER        Type = 0
	NAME             Type = 1
	NUMBER           Type = 2
	STRING           Type = 3
	NEWLINE          Type = 4
	INDENT           Type = 5
	DEDENT           Type = 6
	LPAR             Type = 7
	RPAR             Type = 8
	LSQB             Type = 9
	RSQB             Type = 10
	COLON            Type = 11
	COMMA            Type = 12
	SEMI             Type = 13
	PLUS             Type = 14
	MINUS            Type = 15
	STAR             Type = 16
	SLASH            Type = 17
	VBAR             Type = 18
	AMPER            Type = 19
	LESS             Type = 20
	GREATER          Type = 21
	EQUAL            Type = 22
	DOT              Type = 23
	PERCENT          Type = 24
	LBRACE           Type = 25
	RBRACE           Type = 26
	EQEQUAL          Type = 27
	NOTEQUAL         Type = 28
	LESSEQUAL        Type = 29
	GREATEREQUAL     Type = 30
	TILDE            Type = 31
	CIRCUMFLEX       Type = 32
	LEFTSHIFT        Type = 33
	RIGHTSHIFT       Type = 34
	DOUBLESTAR       Type = 35
	PLUSEQUAL        Type = 36
	MINEQUAL         Type = 37
	STAREQUAL        Type = 38
	SLASHEQUAL       Type = 39
	PERCENTEQUAL     Type = 40
	AMPEREQUAL       Type = 41
	VBAREQUAL        Type = 42
	CIRCUMFLEXEQUAL  Type = 43
	LEFTSHIFTEQUAL   Type = 44
	RIGHTSHIFTEQUAL  Type = 45
	DOUBLESTAREQUAL  Type = 46
	DOUBLESLASH      Type = 47
	DOUBLESLASHEQUAL Type = 48
	AT               Type = 49
	ATEQUAL          Type = 50
	RARROW           Type = 51
	ELLIPSIS         Type = 52
	COLONEQUAL       Type = 53
	EXCLAMATION      Type = 54
	OP               Type = 55
	TYPE_IGNORE      Type = 56
	TYPE_COMMENT     Type = 57
	SOFT_KEYWORD     Type = 58
	FSTRING_START    Type = 59
	FSTRING_MIDDLE   Type = 60
	FSTRING_END      Type = 61
	TSTRING_START    Type = 62
	TSTRING_MIDDLE   Type = 63
	TSTRING_END      Type = 64
	COMMENT          Type = 65
	NL               Type = 66
	ERRORTOKEN       Type = 67
	ENCODING         Type = 68
	NTokens          Type = 69
)

Token kinds, numeric values pinned to CPython's token.h. The ALL_CAPS spellings preserve parity with `token.tok_name` so fixture comparisons line up byte-for-byte.

func (Type) String

func (t Type) String() string

String returns the CPython-compatible token name (e.g. "NAME", "NUMBER"). Unknown values render as "TYPE(n)".

CPython: Lib/token.py tok_name