Documentation
¶
Overview ¶
Package pegen ports the runtime half of cpython 3.14 Parser/pegen.c and pegen.h. The package owns the Parser struct, the token buffer and memo table, the mark/restore backtracking primitives, the keyword tables, and the error tracking surface that the generated parser table calls into.
The generated parser table itself is not in this file. CPython regenerates Parser/parser.c from Grammar/python.gram via Tools/peg_generator/. The Go-targeted equivalent lives in tools/parser_gen/ and emits parser_gen.go in this package.
CPython: Parser/pegen.h, Parser/pegen.c
Index ¶
- Constants
- Variables
- func AugOp(op ast.Operator) ast.Operator
- func ClassDefDecorators(p *Parser, decorators []ast.Expr, classDef ast.Stmt) ast.Stmt
- func CollectCallSeqs(p *Parser, a []ast.Expr, b []*KeywordOrStarred) ast.Expr
- func ConcatenateStrings(p *Parser, parts []ast.Expr) ast.Expr
- func DebugLiteral(text string) *ast.Constant
- func Dispatch(p *Parser, m StartRule) (any, error)
- func EmptyArguments(p *Parser) *ast.Arguments
- func FormattedValueFor(expr ast.Expr, conversion int, format ast.Expr) *ast.FormattedValue
- func FunctionDefDecorators(p *Parser, decorators []ast.Expr, fn ast.Stmt) ast.Stmt
- func GetExprName(e ast.Expr) string
- func GetKeys(pairs []KeyValuePair) []ast.Expr
- func GetValues(pairs []KeyValuePair) []ast.Expr
- func InterpolationFor(expr ast.Expr, srcText string, conversion int, format ast.Expr) *ast.Interpolation
- func JoinIDsWithDot(ids []string) string
- func JoinNamesWithDot(first, second *ast.Name) string
- func JoinedStrFromValues(values []ast.Expr) *ast.JoinedStr
- func MakeArguments(p *Parser, slashWithoutDefault []*ast.Arg, slashWithDefault *SlashWithDefault, ...) *ast.Arguments
- func MakeComprehension(target, iter ast.Expr, ifs []ast.Expr, isAsync bool) *ast.Comprehension
- func MakeDictComp(key, value ast.Expr, gens []*ast.Comprehension) *ast.DictComp
- func MakeGeneratorExp(elt ast.Expr, gens []*ast.Comprehension) *ast.GeneratorExp
- func MakeListComp(elt ast.Expr, gens []*ast.Comprehension) *ast.ListComp
- func MakeSetComp(elt ast.Expr, gens []*ast.Comprehension) *ast.SetComp
- func SeqAppendToEnd[T any](seq []T, a T) []T
- func SeqCountDots(seq []*Token) int
- func SeqFirstItem[T any](seq []T) T
- func SeqFlatten[T any](seqs [][]T) []T
- func SeqInsertInFront[T any](a T, seq []T) []T
- func SeqLastItem[T any](seq []T) T
- func SetExprContext(p *Parser, expr ast.Expr, ctx ast.ExprContext) ast.Expr
- func SingletonSeq[T any](a T) []T
- func TemplateStrFromValues(values []ast.Expr) *ast.TemplateStr
- type CompFor
- type KeyValuePair
- type KeywordOrStarred
- type KeywordToken
- type Location
- type NameDefaultPair
- type Parser
- func (p *Parser) BumpFarthest()
- func (p *Parser) CallInvalid() bool
- func (p *Parser) Debug(v bool)
- func (p *Parser) EnsureImaginary(e ast.Expr) ast.Expr
- func (p *Parser) EnsureReal(e ast.Expr) ast.Expr
- func (p *Parser) EnterFrame() bool
- func (p *Parser) ErrorIndicator() bool
- func (p *Parser) Expect(kind token.Type) *Token
- func (p *Parser) ExpectForced(kind token.Type, expected string) *Token
- func (p *Parser) ExpectName(s string) *Token
- func (p *Parser) ExpectSoftKeyword(kw string) *Token
- func (p *Parser) ExpectToken(kind token.Type) *Token
- func (p *Parser) FarthestPos() int
- func (p *Parser) FeatureVersion() int
- func (p *Parser) Fill() int
- func (p *Parser) FillUntil(n int) int
- func (p *Parser) Flags() int
- func (p *Parser) InsertMemo(mark, ruleType int, node any)
- func (p *Parser) IsMemoized(ruleType int) (any, bool)
- func (p *Parser) LastNonWhitespaceToken() *Token
- func (p *Parser) LeaveFrame()
- func (p *Parser) Lookahead(positive bool, fn func(*Parser) any) bool
- func (p *Parser) LookaheadWithName(positive bool, fn func(*Parser) any, _ string) bool
- func (p *Parser) Mark() int
- func (p *Parser) Peek() *Token
- func (p *Parser) PeekAhead(n int) *Token
- func (p *Parser) PinnedError() *perrors.SyntaxError
- func (p *Parser) RaiseIndentationError(format string, args ...any)
- func (p *Parser) RaiseSyntaxError(format string, args ...any)
- func (p *Parser) RaiseSyntaxErrorKnownLocation(pos perrors.Pos, format string, args ...any)
- func (p *Parser) Reset(m int)
- func (p *Parser) SetCallInvalid(v bool)
- func (p *Parser) SetErrorIndicator(v bool)
- func (p *Parser) SetFeatureVersion(v int)
- func (p *Parser) Span(startMark int) ast.Pos
- func (p *Parser) Tokens() []*Token
- func (p *Parser) UpdateMemo(mark, ruleType int, node any)
- type SlashWithDefault
- type StarEtc
- type StartRule
- type Token
Constants ¶
const ( ConvNone = -1 ConvStr = int('s') ConvRepr = int('r') ConvASCII = int('a') )
Conversion characters as the parser sees them. -1 means absent.
const ( FlagDontImplyDedent = 0x0002 FlagIgnoreCookie = 0x0010 FlagBarryAsBDFL = 0x0020 FlagTypeComments = 0x0040 FlagAllowIncompleteInput = 0x0100 )
Flag bits that mirror PyPARSE_*. The parser respects a subset; the rest are accepted for ABI parity.
CPython: Parser/pegen.h:14 PyPARSE_* flags
const ( Rule_file = 0 Rule_interactive = 1 Rule_eval = 2 Rule_func_type = 3 Rule_statements = 4 Rule_statement = 5 Rule_single_compound_stmt = 6 Rule_statement_newline = 7 Rule_simple_stmts = 8 Rule_simple_stmt = 9 Rule_compound_stmt = 10 Rule_assignment = 11 Rule_annotated_rhs = 12 Rule_augassign = 13 Rule_return_stmt = 14 Rule_raise_stmt = 15 Rule_pass_stmt = 16 Rule_break_stmt = 17 Rule_continue_stmt = 18 Rule_global_stmt = 19 Rule_nonlocal_stmt = 20 Rule_del_stmt = 21 Rule_yield_stmt = 22 Rule_assert_stmt = 23 Rule_import_stmt = 24 Rule_import_name = 25 Rule_import_from = 26 Rule_import_from_targets = 27 Rule_import_from_as_names = 28 Rule_import_from_as_name = 29 Rule_dotted_as_names = 30 Rule_dotted_as_name = 31 Rule_dotted_name = 32 Rule_block = 33 Rule_decorators = 34 Rule_class_def = 35 Rule_class_def_raw = 36 Rule_function_def = 37 Rule_function_def_raw = 38 Rule_params = 39 Rule_parameters = 40 Rule_slash_no_default = 41 Rule_slash_with_default = 42 Rule_star_etc = 43 Rule_kwds = 44 Rule_param_no_default = 45 Rule_param_no_default_star_annotation = 46 Rule_param_with_default = 47 Rule_param_maybe_default = 48 Rule_param = 49 Rule_param_star_annotation = 50 Rule_annotation = 51 Rule_star_annotation = 52 Rule_default = 53 Rule_if_stmt = 54 Rule_elif_stmt = 55 Rule_else_block = 56 Rule_while_stmt = 57 Rule_for_stmt = 58 Rule_with_stmt = 59 Rule_with_item = 60 Rule_try_stmt = 61 Rule_except_block = 62 Rule_except_star_block = 63 Rule_finally_block = 64 Rule_match_stmt = 65 Rule_subject_expr = 66 Rule_case_block = 67 Rule_guard = 68 Rule_patterns = 69 Rule_pattern = 70 Rule_as_pattern = 71 Rule_or_pattern = 72 Rule_closed_pattern = 73 Rule_literal_pattern = 74 Rule_literal_expr = 75 Rule_complex_number = 76 Rule_signed_number = 77 Rule_signed_real_number = 78 Rule_real_number = 79 Rule_imaginary_number = 80 Rule_capture_pattern = 81 Rule_pattern_capture_target = 82 Rule_wildcard_pattern = 83 Rule_value_pattern = 84 Rule_attr = 85 Rule_name_or_attr = 86 Rule_group_pattern = 87 Rule_sequence_pattern = 88 Rule_open_sequence_pattern = 89 Rule_maybe_sequence_pattern = 90 Rule_maybe_star_pattern = 91 Rule_star_pattern = 92 Rule_mapping_pattern = 93 Rule_items_pattern = 94 Rule_key_value_pattern = 95 Rule_double_star_pattern = 96 Rule_class_pattern = 97 Rule_positional_patterns = 98 Rule_keyword_patterns = 99 Rule_keyword_pattern = 100 Rule_type_alias = 101 Rule_type_params = 102 Rule_type_param_seq = 103 Rule_type_param = 104 Rule_type_param_bound = 105 Rule_type_param_default = 106 Rule_type_param_starred_default = 107 Rule_expressions = 108 Rule_expression = 109 Rule_yield_expr = 110 Rule_star_expressions = 111 Rule_star_expression = 112 Rule_star_named_expressions = 113 Rule_star_named_expression = 114 Rule_assignment_expression = 115 Rule_named_expression = 116 Rule_disjunction = 117 Rule_conjunction = 118 Rule_inversion = 119 Rule_comparison = 120 Rule_compare_op_bitwise_or_pair = 121 Rule_eq_bitwise_or = 122 Rule_noteq_bitwise_or = 123 Rule_lte_bitwise_or = 124 Rule_lt_bitwise_or = 125 Rule_gte_bitwise_or = 126 Rule_gt_bitwise_or = 127 Rule_notin_bitwise_or = 128 Rule_in_bitwise_or = 129 Rule_isnot_bitwise_or = 130 Rule_is_bitwise_or = 131 Rule_bitwise_or = 132 Rule_bitwise_xor = 133 Rule_bitwise_and = 134 Rule_shift_expr = 135 Rule_sum = 136 Rule_term = 137 Rule_factor = 138 Rule_power = 139 Rule_await_primary = 140 Rule_primary = 141 Rule_slices = 142 Rule_slice = 143 Rule_atom = 144 Rule_group = 145 Rule_lambdef = 146 Rule_lambda_params = 147 Rule_lambda_parameters = 148 Rule_lambda_slash_no_default = 149 Rule_lambda_slash_with_default = 150 Rule_lambda_star_etc = 151 Rule_lambda_kwds = 152 Rule_lambda_param_no_default = 153 Rule_lambda_param_with_default = 154 Rule_lambda_param_maybe_default = 155 Rule_lambda_param = 156 Rule_fstring_middle = 157 Rule_fstring_replacement_field = 158 Rule_fstring_conversion = 159 Rule_fstring_full_format_spec = 160 Rule_fstring_format_spec = 161 Rule_fstring = 162 Rule_tstring_format_spec_replacement_field = 163 Rule_tstring_format_spec = 164 Rule_tstring_full_format_spec = 165 Rule_tstring_replacement_field = 166 Rule_tstring_middle = 167 Rule_tstring = 168 Rule_string = 169 Rule_strings = 170 Rule_list = 171 Rule_tuple = 172 Rule_set = 173 Rule_dict = 174 Rule_double_starred_kvpairs = 175 Rule_double_starred_kvpair = 176 Rule_kvpair = 177 Rule_for_if_clauses = 178 Rule_for_if_clause = 179 Rule_listcomp = 180 Rule_setcomp = 181 Rule_genexp = 182 Rule_dictcomp = 183 Rule_arguments = 184 Rule_args = 185 Rule_kwargs = 186 Rule_starred_expression = 187 Rule_kwarg_or_starred = 188 Rule_kwarg_or_double_starred = 189 Rule_star_targets = 190 Rule_star_targets_list_seq = 191 Rule_star_targets_tuple_seq = 192 Rule_star_target = 193 Rule_target_with_star_atom = 194 Rule_star_atom = 195 Rule_single_target = 196 Rule_single_subscript_attribute_target = 197 Rule_t_primary = 198 Rule_t_lookahead = 199 Rule_del_targets = 200 Rule_del_target = 201 Rule_del_t_atom = 202 Rule_type_expressions = 203 Rule_func_type_comment = 204 Rule_invalid_arguments = 205 Rule_invalid_kwarg = 206 Rule_expression_without_invalid = 207 Rule_invalid_legacy_expression = 208 Rule_invalid_type_param = 209 Rule_invalid_expression = 210 Rule_invalid_named_expression = 211 Rule_invalid_assignment = 212 Rule_invalid_ann_assign_target = 213 Rule_invalid_del_stmt = 214 Rule_invalid_block = 215 Rule_invalid_comprehension = 216 Rule_invalid_dict_comprehension = 217 Rule_invalid_parameters = 218 Rule_invalid_default = 219 Rule_invalid_star_etc = 220 Rule_invalid_kwds = 221 Rule_invalid_parameters_helper = 222 Rule_invalid_lambda_parameters = 223 Rule_invalid_lambda_parameters_helper = 224 Rule_invalid_lambda_star_etc = 225 Rule_invalid_lambda_kwds = 226 Rule_invalid_double_type_comments = 227 Rule_invalid_with_item = 228 Rule_invalid_for_if_clause = 229 Rule_invalid_for_target = 230 Rule_invalid_group = 231 Rule_invalid_import = 232 Rule_invalid_dotted_as_name = 233 Rule_invalid_import_from_as_name = 234 Rule_invalid_import_from_targets = 235 Rule_invalid_with_stmt = 236 Rule_invalid_with_stmt_indent = 237 Rule_invalid_try_stmt = 238 Rule_invalid_except_stmt = 239 Rule_invalid_except_star_stmt = 240 Rule_invalid_finally_stmt = 241 Rule_invalid_except_stmt_indent = 242 Rule_invalid_except_star_stmt_indent = 243 Rule_invalid_match_stmt = 244 Rule_invalid_case_block = 245 Rule_invalid_as_pattern = 246 Rule_invalid_class_pattern = 247 Rule_invalid_class_argument_pattern = 248 Rule_invalid_if_stmt = 249 Rule_invalid_elif_stmt = 250 Rule_invalid_else_stmt = 251 Rule_invalid_while_stmt = 252 Rule_invalid_for_stmt = 253 Rule_invalid_def_raw = 254 Rule_invalid_class_def_raw = 255 Rule_invalid_double_starred_kvpairs = 256 Rule_invalid_kvpair = 257 Rule_invalid_starred_expression_unpacking = 258 Rule_invalid_starred_expression = 259 Rule_invalid_fstring_replacement_field = 260 Rule_invalid_fstring_conversion_character = 261 Rule_invalid_tstring_replacement_field = 262 Rule_invalid_tstring_conversion_character = 263 Rule_invalid_string_tstring_concat = 264 Rule_invalid_arithmetic = 265 Rule_invalid_factor = 266 Rule_invalid_type_params = 267 Rule__loop0_1 = 268 Rule__loop1_2 = 269 Rule__gather_3 = 270 Rule__group_4 = 271 Rule__group_5 = 272 Rule__group_6 = 273 Rule__group_7 = 274 Rule__group_8 = 275 Rule__rhs_9 = 276 Rule__group_10 = 277 Rule__loop1_11 = 278 Rule__rhs_12 = 279 Rule__gather_13 = 280 Rule__group_14 = 281 Rule__rhs_15 = 282 Rule__loop0_16 = 283 Rule__loop1_17 = 284 Rule__gather_18 = 285 Rule__rhs_19 = 286 Rule__gather_20 = 287 Rule__loop1_21 = 288 Rule__rhs_22 = 289 Rule__rhs_23 = 290 Rule__loop0_24 = 291 Rule__loop0_25 = 292 Rule__loop1_26 = 293 Rule__loop1_27 = 294 Rule__loop0_28 = 295 Rule__loop1_29 = 296 Rule__gather_30 = 297 Rule__group_31 = 298 Rule__loop1_32 = 299 Rule__loop1_33 = 300 Rule__loop1_34 = 301 Rule__gather_35 = 302 Rule__group_36 = 303 Rule__group_37 = 304 Rule__group_38 = 305 Rule__gather_39 = 306 Rule__gather_40 = 307 Rule__group_41 = 308 Rule__gather_42 = 309 Rule__gather_43 = 310 Rule__gather_44 = 311 Rule__loop1_45 = 312 Rule__loop1_46 = 313 Rule__gather_47 = 314 Rule__loop1_48 = 315 Rule__loop1_49 = 316 Rule__loop1_50 = 317 Rule__gather_51 = 318 Rule__rhs_52 = 319 Rule__group_53 = 320 Rule__group_54 = 321 Rule__group_55 = 322 Rule__group_56 = 323 Rule__loop0_57 = 324 Rule__loop0_58 = 325 Rule__loop1_59 = 326 Rule__loop1_60 = 327 Rule__loop0_61 = 328 Rule__loop1_62 = 329 Rule__loop0_63 = 330 Rule__loop0_64 = 331 Rule__loop0_65 = 332 Rule__loop0_66 = 333 Rule__loop1_67 = 334 Rule__loop1_68 = 335 Rule__rhs_69 = 336 Rule__gather_70 = 337 Rule__loop1_71 = 338 Rule__loop0_72 = 339 Rule__group_73 = 340 Rule__gather_74 = 341 Rule__rhs_75 = 342 Rule__gather_76 = 343 Rule__gather_77 = 344 Rule__loop0_78 = 345 Rule__gather_79 = 346 Rule__loop1_80 = 347 Rule__group_81 = 348 Rule__gather_82 = 349 Rule__gather_83 = 350 Rule__group_84 = 351 Rule__group_85 = 352 Rule__gather_86 = 353 Rule__rhs_87 = 354 Rule__group_88 = 355 Rule__group_89 = 356 Rule__group_90 = 357 Rule__group_91 = 358 Rule__loop1_92 = 359 Rule__group_93 = 360 Rule__group_94 = 361 Rule__group_95 = 362 Rule__group_96 = 363 Rule__group_97 = 364 Rule__loop0_98 = 365 Rule__loop0_99 = 366 Rule__group_100 = 367 Rule__group_101 = 368 Rule__group_102 = 369 Rule__group_103 = 370 Rule__group_104 = 371 Rule__group_105 = 372 Rule__group_106 = 373 Rule__group_107 = 374 Rule__group_108 = 375 Rule__gather_109 = 376 Rule__group_110 = 377 Rule__group_111 = 378 Rule__group_112 = 379 Rule__group_113 = 380 Rule__gather_114 = 381 Rule__group_115 = 382 Rule__gather_116 = 383 Rule__gather_117 = 384 Rule__group_118 = 385 Rule__loop0_119 = 386 Rule__rhs_120 = 387 Rule__rhs_121 = 388 Rule__group_122 = 389 Rule__rhs_123 = 390 Rule__rhs_124 = 391 Rule__rhs_125 = 392 Rule__group_126 = 393 Rule__group_127 = 394 Rule__group_128 = 395 Rule__rhs_129 = 396 Rule__group_130 = 397 Rule__group_131 = 398 Rule__group_132 = 399 Rule__group_133 = 400 Rule__group_134 = 401 Rule__group_135 = 402 Rule__group_136 = 403 Rule__group_137 = 404 Rule__group_138 = 405 Rule__group_139 = 406 Rule__group_140 = 407 Rule__group_141 = 408 Rule__group_142 = 409 Rule__group_143 = 410 Rule__group_144 = 411 Rule__group_145 = 412 Rule__group_146 = 413 Rule__group_147 = 414 Rule__group_148 = 415 Rule__group_149 = 416 Rule__group_150 = 417 Rule__loop0_151 = 418 Rule__group_152 = 419 Rule__group_153 = 420 Rule__group_154 = 421 Rule__group_155 = 422 Rule__rhs_156 = 423 )
Generated rule ids. Dense small ints in declaration order, then artificial helper rules. Stable across regenerations as long as python.gram does not insert, reorder, or rephrase rules in ways that change the artificial-rule deduplication keys.
const MaxRecursionDepth = 6000
MaxRecursionDepth bounds left-recursive rule depth. The generated parser bumps p.level on each rule entry and decrements on exit; trip the guard with StackOverflow().
CPython: Parser/pegen.h:60 _PyOS_RecursionLimit
Variables ¶
var ErrParserNotImplemented = errors.New("pegen: generated rule bodies not yet emitted")
ErrParserNotImplemented is returned by Dispatch while the per-rule emitter still produces placeholder action results. The real AST surface lands with the action translator (M6).
var GeneratedRuleNames = []string{}/* 424 elements not displayed */
GeneratedRuleNames is the parallel name table.
var HardKeywords = []string{
"False",
"None",
"True",
"and",
"as",
"assert",
"async",
"await",
"break",
"class",
"continue",
"def",
"del",
"elif",
"else",
"except",
"finally",
"for",
"from",
"global",
"if",
"import",
"in",
"is",
"lambda",
"nonlocal",
"not",
"or",
"pass",
"raise",
"return",
"try",
"while",
"with",
"yield",
}
HardKeywords are the single-quoted literals from python.gram.
var SoftKeywords = []string{
"!",
"/",
"_",
"case",
"match",
"type",
}
SoftKeywords are the double-quoted literals from python.gram.
Functions ¶
func AugOp ¶
AugOp wraps an Operator value so the rule action that builds an AugAssign can carry the operator separately from the target / value pair before assembling the statement.
CPython: Parser/action_helpers.c:716 _PyPegen_augoperator
func ClassDefDecorators ¶ added in v0.6.0
ClassDefDecorators returns a fresh ClassDef equivalent to classDef but with decorators stamped in.
CPython: Parser/action_helpers.c:756 _PyPegen_class_def_decorators
func CollectCallSeqs ¶
CollectCallSeqs builds the dummy-Call carrier the args rule returns. The surrounding primary alt extracts .Args and .Keywords and rebuilds the real Call with the actual callee.
args[expr_ty]:
| a=','.starred_or_expr+ b=[',' k=kwargs {k}] {
_PyPegen_collect_call_seqs(p, a, b, EXTRA) }
b is nil when the source has no kwargs tail. When present, it is a flat sequence of *KeywordOrStarred whose starred elements append to the positional list and whose keyword elements become the keyword list.
CPython: Parser/action_helpers.c:1129 _PyPegen_collect_call_seqs
func ConcatenateStrings ¶
ConcatenateStrings folds a sequence of adjacent string / fstring pieces into a single AST node. The shape mirrors CPython exactly:
- all-plain unicode pieces collapse to one Constant string;
- all-bytes pieces collapse to one Constant []byte;
- mixing bytes with anything non-bytes pins the parser's error indicator (the caller surfaces "cannot mix bytes and nonbytes literals");
- any f-string piece flips the result into a JoinedStr whose values are the flattened-and-folded element list.
EXTRA position arguments are dropped at the action-translator layer, so the assembled nodes use ast.NoPos.
CPython: Parser/action_helpers.c:1860 _PyPegen_concatenate_strings
func DebugLiteral ¶
DebugLiteral builds the literal "expr=" Constant the {expr=} debug form prepends before the formatted value.
CPython: Parser/action_helpers.c:1780 fstring_debug_literal
func Dispatch ¶
Dispatch picks the entry-point rule for m and runs it. Mirrors CPython's _PyPegen_run_parser: try once with call_invalid_rules off, retry once with it on if the first pass missed, then surface the rule result. If the second pass also misses, ErrParserNotImplemented rides through so callers can fall back to the fixture path until the action surface fully types up.
CPython: Parser/pegen.c:946 _PyPegen_run_parser
func EmptyArguments ¶
EmptyArguments returns the all-empty Arguments node CPython lifts when a function has no parameter list.
CPython: Parser/action_helpers.c:686 _PyPegen_empty_arguments
func FormattedValueFor ¶
FormattedValueFor builds a FormattedValue from the (expr, conversion, format_spec) triple a {expr!c:fmt} rule produces.
CPython: Parser/action_helpers.c:1748 _PyPegen_formatted_value
func FunctionDefDecorators ¶
FunctionDefDecorators returns a fresh FunctionDef / AsyncFunctionDef equivalent to fn but with decorators stamped in. CPython rebuilds the node so the original (decorator-free) value the function_def_raw rule produced stays untouched in the arena.
CPython: Parser/action_helpers.c:727 _PyPegen_function_def_decorators
func GetExprName ¶
GetExprName returns the human-readable phrase used in "cannot assign to %s" diagnostics for an expression.
CPython: Parser/action_helpers.c:1259 _PyPegen_get_expr_name
func GetKeys ¶
func GetKeys(pairs []KeyValuePair) []ast.Expr
GetKeys flattens the key side of a list of (key, value) pairs.
CPython: Parser/action_helpers.c:354 _PyPegen_get_keys
func GetValues ¶
func GetValues(pairs []KeyValuePair) []ast.Expr
GetValues flattens the value side of a (key, value) pair list.
CPython: Parser/action_helpers.c:370 _PyPegen_get_values
func InterpolationFor ¶
func InterpolationFor(expr ast.Expr, srcText string, conversion int, format ast.Expr) *ast.Interpolation
InterpolationFor is the t-string equivalent. The Str field carries the source text of the expression, used at runtime by Template.
CPython: Parser/action_helpers.c (PEP 750)
func JoinIDsWithDot ¶
JoinIDsWithDot is the n-ary form, used by the import-name rule. CPython composes by calling join_names_with_dot in a fold; gopy hands the slice over directly because Go has no asdl_seq.
CPython: Parser/action_helpers.c:181 (n-ary fold pattern)
func JoinNamesWithDot ¶
JoinNamesWithDot turns two ast.Name expressions into "a.b". The generated parser chains this for dotted import names.
CPython: Parser/action_helpers.c:181 _PyPegen_join_names_with_dot
func JoinedStrFromValues ¶
JoinedStrFromValues lifts a slice of literal/expression parts into a JoinedStr node. Adjacent plain Constant strings are folded to keep the AST flat, mirroring _PyPegen_concatenate_strings' post-pass.
CPython: Parser/action_helpers.c:1860 _PyPegen_concatenate_strings
func MakeArguments ¶
func MakeArguments( p *Parser, slashWithoutDefault []*ast.Arg, slashWithDefault *SlashWithDefault, plainNames []*ast.Arg, namesWithDefault []*NameDefaultPair, se *StarEtc, ) *ast.Arguments
MakeArguments folds the parameter-list bundle into a single Arguments node.
CPython: Parser/action_helpers.c:643 _PyPegen_make_arguments
func MakeComprehension ¶
MakeComprehension is the per-clause builder. The for_if_clause rule emits one of these and the comprehension rule glues several together into the Generators slice.
CPython: Parser/action_helpers.c:885 _PyPegen_comprehension_for_clause
func MakeDictComp ¶
MakeDictComp builds a DictComp node from the (key, value) head and the generator list.
CPython: Parser/action_helpers.c:945 _PyPegen_make_dictcomp
func MakeGeneratorExp ¶
func MakeGeneratorExp(elt ast.Expr, gens []*ast.Comprehension) *ast.GeneratorExp
MakeGeneratorExp builds a GeneratorExp node.
CPython: Parser/action_helpers.c:955 _PyPegen_make_generatorexp
func MakeListComp ¶
MakeListComp wraps an element expression and the for-clause list into a ListComp node.
CPython: Parser/action_helpers.c:925 _PyPegen_make_listcomp
func MakeSetComp ¶
MakeSetComp builds a SetComp node.
CPython: Parser/action_helpers.c:935 _PyPegen_make_setcomp
func SeqAppendToEnd ¶
func SeqAppendToEnd[T any](seq []T, a T) []T
SeqAppendToEnd returns a new slice with a appended to seq.
CPython: Parser/action_helpers.c:114 _PyPegen_seq_append_to_end
func SeqCountDots ¶
SeqCountDots counts the leading DOT/ELLIPSIS tokens that the `from ... import` rule accepts. ELLIPSIS counts as three dots.
CPython: Parser/action_helpers.c:198 _PyPegen_seq_count_dots
func SeqFirstItem ¶
func SeqFirstItem[T any](seq []T) T
SeqFirstItem returns seq[0] or the zero value if empty.
CPython: Parser/action_helpers.c:172 _PyPegen_seq_first_item
func SeqFlatten ¶
func SeqFlatten[T any](seqs [][]T) []T
SeqFlatten concatenates a slice of slices.
CPython: Parser/action_helpers.c:131 _PyPegen_seq_flatten
func SeqInsertInFront ¶
func SeqInsertInFront[T any](a T, seq []T) []T
SeqInsertInFront returns a new slice with a prepended to seq. The C source allocates a fresh asdl_seq; the Go version makes a copy for the same reason (caller may still hold seq).
CPython: Parser/action_helpers.c:97 _PyPegen_seq_insert_in_front
func SeqLastItem ¶
func SeqLastItem[T any](seq []T) T
SeqLastItem returns the last item or the zero value if empty.
CPython: Parser/action_helpers.c:163 _PyPegen_seq_last_item
func SetExprContext ¶
SetExprContext returns a copy of expr with ctx as the context. Name, Tuple, List, Subscript, Attribute and Starred are rewritten; every other expression kind is returned unchanged. Tuple and List recurse through their element list via setSeqContext; Starred recurses into its inner value.
CPython: Parser/action_helpers.c:309 _PyPegen_set_expr_context
func SingletonSeq ¶
func SingletonSeq[T any](a T) []T
SingletonSeq wraps one element into a one-element slice. The generated parser uses this to lift a single rule result into the list shape that the next rule expects.
CPython: Parser/action_helpers.c:80 _PyPegen_singleton_seq
func TemplateStrFromValues ¶
func TemplateStrFromValues(values []ast.Expr) *ast.TemplateStr
TemplateStrFromValues is the t-string counterpart.
CPython: Parser/action_helpers.c (PEP 750)
Types ¶
type CompFor ¶
type CompFor = ast.Comprehension
CompFor is the (target, iter, ifs, is_async) tuple a `for` clause produces inside a comprehension.
CPython: Parser/pegen.h:127 CompFor
type KeyValuePair ¶
KeyValuePair is the parser-side struct that carries a (key, value) pair through dict literal and dict-comprehension rules.
CPython: Parser/pegen.h:96 KeyValuePair
type KeywordOrStarred ¶
KeywordOrStarred is the union the kwarg_or_starred / kwarg_or_double_starred rules produce: each entry carries either a positional / *args expression (IsKeyword false) or a (name, value) keyword (IsKeyword true). The collect-call-seqs path splits the flat list back into the asdl_expr_seq / asdl_keyword_seq pair Call expects.
CPython: Parser/pegen.h:91 KeywordOrStarred
type KeywordToken ¶
KeywordToken pairs a keyword string with the NAME-derived token kind the generated parser table compares against.
CPython: Parser/pegen.h:48 KeywordToken
type Location ¶
Location pins a (start, end) source span. The generated parser uses this to attach source positions to AST nodes.
CPython: Parser/pegen.h:63 location
type NameDefaultPair ¶
NameDefaultPair is the (arg, default) tuple a regular parameter rule emits. nameDefaultPair is the constructor; the parser action wraps the (arg, value, type-comment) triple into a fresh pair.
CPython: Parser/pegen.h:111 NameDefaultPair
type Parser ¶
type Parser struct {
// contains filtered or unexported fields
}
Parser is the runtime state. The generated table threads through this struct via mark/restore and the consume helpers.
CPython: Parser/pegen.h:70 Parser
func New ¶
New builds a Parser around an already-configured lexer State. The token buffer is filled lazily as the generated parser advances.
CPython: Parser/pegen.c:1024 _PyPegen_Parser_New
func (*Parser) BumpFarthest ¶
func (p *Parser) BumpFarthest()
BumpFarthest is called by the generated parser at the head of every rule alt that consumes a token. Keeps p.farthestPos at the deepest mark seen so far.
func (*Parser) CallInvalid ¶
CallInvalid reports the second-pass flag.
func (*Parser) Debug ¶
Debug enables verbose rule-trace output. The generated parser reads this in the dispatch loop; gopy uses it identically.
CPython: Parser/pegen.c:973 _PyPegen_set_debug
func (*Parser) EnsureImaginary ¶
EnsureImaginary is the matching check for the imaginary side.
CPython: Parser/action_helpers.c:843 _PyPegen_ensure_imaginary
func (*Parser) EnsureReal ¶
EnsureReal mirrors the runtime check that the imaginary side of a complex-literal sum (`1 + 2j`) is a real numeric Constant. It returns nil and sets the error indicator if the check fails.
CPython: Parser/action_helpers.c:853 _PyPegen_ensure_real
func (*Parser) EnterFrame ¶
EnterFrame is called at the head of every generated rule. It returns false when the depth guard trips, in which case the rule must return nil and stop walking.
CPython: Parser/pegen.c:902 _PyPegen_check_recursion_limit
func (*Parser) ErrorIndicator ¶
ErrorIndicator reports whether the parser has hit a fatal error.
func (*Parser) Expect ¶
Expect consumes the token at the mark if it matches kind, returning the token; otherwise returns nil and leaves mark unchanged.
A NAME token whose bytes are a hard keyword is refused: CPython's lexer assigns those a per-keyword token type so the generic NAME alts never see them. gopy's lexer emits NAME for every identifier, so the discrimination has to happen at the parser layer instead. ExpectName / ExpectSoftKeyword bypass this filter; they match by bytes and need to accept keyword tokens.
CPython: Parser/pegen.c:296 _PyPegen_expect_token
func (*Parser) ExpectForced ¶
ExpectForced advances past a token of kind kind. If the next token does not match, it raises a SyntaxError with the "expected '%s'" template and trips the error indicator.
CPython: Parser/pegen.c:441 _PyPegen_expect_forced_token
func (*Parser) ExpectName ¶
ExpectName consumes a NAME token whose bytes match s.
CPython: Parser/pegen.c:268 _PyPegen_expect_keyword
func (*Parser) ExpectSoftKeyword ¶
ExpectSoftKeyword advances past a NAME whose body matches kw. Soft keywords (match, case, type, _) are not in the keyword table, so the generated parser uses this to recognize them by content rather than token kind.
CPython: Parser/pegen.c:425 _PyPegen_expect_soft_keyword
func (*Parser) ExpectToken ¶
ExpectToken is the kind-only form CPython emits as expect_token. Returns the token on match (advancing mark) or nil on miss.
CPython: Parser/pegen.c:296 _PyPegen_expect_token
func (*Parser) FarthestPos ¶
FarthestPos reports the deepest token index any rule has reached during this parse. Diagnostic; used by error pinning to decide whether a fresh error replaces the stored one.
CPython: Parser/pegen.h farthest_pos
func (*Parser) FeatureVersion ¶
FeatureVersion pins the from __future__ minor version the parser is targeting. Defaults to current; rules that gate on PEP-introduced syntax read this field.
CPython: Parser/pegen.h:88 feature_version
func (*Parser) FillUntil ¶
FillUntil grows the token buffer until it covers `mark+n` and returns the resulting fill index. Used by Lookahead and the invalid-rule replay machinery.
CPython: Parser/pegen.c:69 _PyPegen_fill_until
func (*Parser) InsertMemo ¶
InsertMemo caches a parse result. mark is the position at which the rule started; the call site captures that before invoking the rule, then passes it back here once a node was produced.
CPython: Parser/pegen.c:80 _PyPegen_insert_memo
func (*Parser) IsMemoized ¶
IsMemoized checks whether the rule with id ruleType has already been parsed at the current mark. If yes, advances mark past the cached match and returns the cached node and true. If no, returns nil and false. Returns false also on EOF.
CPython: Parser/pegen.c:349 _PyPegen_is_memoized
func (*Parser) LastNonWhitespaceToken ¶
LastNonWhitespaceToken returns the most recently seen token that is not a NEWLINE, NL, INDENT, or DEDENT. Used by error messages that want to point at the real preceding token.
CPython: Parser/pegen.c:469 _PyPegen_get_last_nonnwhitespace_token
func (*Parser) LeaveFrame ¶
func (p *Parser) LeaveFrame()
LeaveFrame matches EnterFrame; safe to call even when the guard tripped because the depth counter is symmetric.
func (*Parser) Lookahead ¶
Lookahead runs fn at the current mark, restores mark, and returns whether the result matches the requested polarity. Positive-true reports whether fn produced a non-nil node.
fn's any return is treated as nil when it carries a typed-nil pointer (e.g. `*Token(nil)` returned from ExpectToken). Without the unwrap the interface header keeps the type word and `!= nil` reports a spurious match, which trips negative-lookahead alts.
CPython: Parser/pegen.c:392 _PyPegen_lookahead
func (*Parser) LookaheadWithName ¶
LookaheadWithName runs fn at the current mark, restores mark, and returns whether the result matches the requested polarity. The helper is the named-result variant the generated parser emits when a positive-lookahead block has a label; the body is identical to Lookahead, the name is propagated by the generator into the surrounding rule's diagnostic plumbing.
CPython: Parser/pegen.c:402 _PyPegen_lookahead_with_name
func (*Parser) Mark ¶
Mark returns the current parse position. Pair with Reset.
CPython: Parser/pegen.c:43 _PyPegen_seek
func (*Parser) Peek ¶
Peek returns the token at the current mark, filling the buffer if needed. Returns nil at EOF.
CPython: Parser/pegen.c:80 _PyPegen_get_token
func (*Parser) PeekAhead ¶
PeekAhead returns the token n positions past the current mark, filling the buffer if needed. PeekAhead(0) is the same as Peek().
CPython: Parser/pegen.c:128 _PyPegen_lookahead_with_token
func (*Parser) PinnedError ¶
func (p *Parser) PinnedError() *perrors.SyntaxError
PinnedError returns the SyntaxError currently pinned on the parser, or nil if none. The driver calls this once parsing fails to convert the indicator into a return value.
CPython: Parser/pegen.c:1136 _PyPegen_run_parser error promotion
func (*Parser) RaiseIndentationError ¶
RaiseIndentationError pins an IndentationError. Mirrors RAISE_INDENTATION_ERROR in pegen.h.
CPython: Parser/pegen.h RAISE_INDENTATION_ERROR
func (*Parser) RaiseSyntaxError ¶
RaiseSyntaxError pins a SyntaxError at the current parse position. The generated parser calls this from a failing rule body. The "farthest reached" heuristic keeps the latest, deepest record; earlier shallower errors are dropped.
CPython: Parser/pegen_errors.c:228 _PyPegen_raise_error
func (*Parser) RaiseSyntaxErrorKnownLocation ¶
RaiseSyntaxErrorKnownLocation pins a SyntaxError at a caller-given span. Used by action helpers that already know where they want the caret to land (invalid LHS, "did you mean :=", paren mismatch).
CPython: Parser/pegen_errors.c:317 _PyPegen_raise_error_known_location
func (*Parser) SetCallInvalid ¶
SetCallInvalid flips the second-pass flag the generated parser reads when invalid_ rules should fire.
CPython: Parser/pegen.c:946 _PyPegen_run_parser
func (*Parser) SetErrorIndicator ¶
SetErrorIndicator pins the error flag. The action_helpers layer flips this when it constructs a SyntaxError.
func (*Parser) SetFeatureVersion ¶
SetFeatureVersion pins the from __future__ minor version.
func (*Parser) Span ¶ added in v0.12.3
Span returns the source location covering tokens [startMark, mark). The end mark is the parser's current position; the last consumed token is p.tokens[mark-1]. Returns ast.NoPos when startMark is out of range or the run is empty.
CPython: Tools/peg_generator/pegen/c_generator.py EXTRA macro (start_lineno / start_col_offset / end_lineno / end_col_offset captured from the parser's mark/end_mark).
func (*Parser) Tokens ¶
Tokens returns a snapshot of the filled token buffer. The generated parser only needs Peek/Expect; this is here for debug printing and error reporting.
func (*Parser) UpdateMemo ¶
UpdateMemo overwrites an existing cache entry for the same rule or inserts a new one. Used by left-recursive rule bodies.
CPython: Parser/pegen.c:97 _PyPegen_update_memo
type SlashWithDefault ¶
type SlashWithDefault struct {
PlainNames []*ast.Arg
NamesWithDefaults []*NameDefaultPair
}
SlashWithDefault carries the (plain-names, names-with-defaults) pair the slash_with_default parameter rule emits. The bundle holds position-only parameters; the surrounding makeArguments folds it into Arguments.posonlyargs / Arguments.defaults.
CPython: Parser/pegen.h:116 SlashWithDefault
type StarEtc ¶
type StarEtc struct {
Vararg *ast.Arg
Kwonlyargs []*NameDefaultPair
Kwarg *ast.Arg
}
StarEtc holds the (*vararg, kwonlyargs, **kwarg) tail of a parameter list. kwonlyargs is a slice of NameDefaultPair so that a kwonly arg can carry its own default.
CPython: Parser/pegen.h:121 StarEtc
type StartRule ¶
type StartRule int
StartRule selects the entry point of the generated parser table. Mirrors the Py_*_input constants.
CPython: Include/compile.h Py_file_input / Py_eval_input / Py_single_input
type Token ¶
type Token struct {
Type token.Type
Bytes []byte
Level int
Lineno int
ColOff int
EndLine int
EndCol int
Metadata []byte
// contains filtered or unexported fields
}
Token is the parser's view of a lexer token. It pins the metadata the action_helpers layer needs for error messages and AST building.
CPython: Parser/pegen.h:39 Token
Source Files