preprocess

Documentation

Overview

Package preprocess implements the Pawn preprocessor: directive parsing, object-like and function-like macro expansion, conditional-compilation branch tracking, and source maps from expanded tokens back to the original source.

Run tokenizes and processes source once, returning a Result that keeps three views alive simultaneously, per the ecosystem's "preprocessing is a first-class language stage" principle:

• Original: Result.OriginalTokens, the unmodified lexer output.
• Active/inactive: Result.Branches records every #if/#elseif/#else branch with its body span and whether it was selected.
• Expanded: Result.ExpandedTokens, the macro-expanded, directive-free token stream ready for github.com/pawnkit/pawn-parser's ParseTokensCompact. Expanded tokens carry a token.Origin chain (spelling location vs. expansion location, in the Clang sense) usable to map a diagnostic on expanded code back to where the responsible macro was invoked.

#include/#tryinclude resolution is delegated to a caller-supplied IncludeResolver so this package never touches the filesystem or a project model directly; see docs/architecture.md for the ownership rationale.

Index

• func ReuseCompatibleContext(ctx context.Context, src []byte, uri string, cache *TokenCache, ...) (*Result, CompatibleEdit, bool, error)
• type Branch
• type ByteRange
• type Code
• type CompatibleEdit
• type Diagnostic
  • func (d Diagnostic) ToCore(file source.FileID) diagnostic.Diagnostic
• type DirectiveKind
  • func (k DirectiveKind) String() string
• type FileInfo
• type Include
• type IncludeResolver
• type Macro
  • func (m Macro) ReplacementCallable() (string, bool)
• type MacroInvocation
• type MacroKind
  • func (k MacroKind) String() string
• type MapResolver
  • func (m MapResolver) Resolve(_, path string, _ bool) ([]byte, string, bool)
• type Options
• type Result
  • func ReuseTriviaContext(ctx context.Context, src []byte, uri string, cache *TokenCache, ...) (*Result, bool, error)
  • func Run(src []byte, opts Options) *Result
  • func RunContext(ctx context.Context, src []byte, opts Options) (*Result, error)
  • func (r *Result) ToCoreDiagnostics(root source.FileID) []diagnostic.Diagnostic
  • func (r *Result) ToRegistryDiagnostics(registry *source.Registry, fallback source.FileID) []diagnostic.Diagnostic
• type TokenCache
  • func NewTokenCache() *TokenCache

Functions

func ReuseCompatibleContext

func ReuseCompatibleContext(
	ctx context.Context,
	src []byte,
	uri string,
	cache *TokenCache,
	previous *Result,
) (*Result, CompatibleEdit, bool, error)

ReuseCompatibleContext retains the dependency graph for a local token edit. Expanded source still contains the previous local body.

Types

type Branch

type Branch struct {
	File          uint32
	Directive     DirectiveKind
	Depth         int
	DirectiveSpan ByteRange
	ConditionSpan ByteRange // zero for #else
	BodySpan      ByteRange
	Active        bool
	Evaluated     bool // false when short-circuited by an inactive parent
}

Branch is one #if/#elseif/#else region, preserving both its own extent and whether it was selected, so callers can reconstruct active and inactive views of the original source without pawn-analysis discarding anything.

type ByteRange

type ByteRange struct {
	Start int
	End   int
}

ByteRange is a half-open source byte range within one file (see Diagnostic.File).

type Code

type Code string

Code is a stable, machine-readable preprocessor diagnostic identifier.

const (
	CodeUnterminatedConditional Code = "preprocess/unterminated-conditional"
	CodeUnmatchedElseif         Code = "preprocess/unmatched-elseif"
	CodeUnmatchedElse           Code = "preprocess/unmatched-else"
	CodeUnmatchedEndif          Code = "preprocess/unmatched-endif"
	CodeConditionalDepthLimit   Code = "preprocess/conditional-depth-limit"
	CodeUnresolvableCondition   Code = "preprocess/unresolvable-condition"
	CodeUnknownDirective        Code = "preprocess/unknown-directive"
	CodeMalformedDefine         Code = "preprocess/malformed-define"
	CodeMacroArgumentMismatch   Code = "preprocess/macro-argument-mismatch"
	CodeUnterminatedInvocation  Code = "preprocess/unterminated-macro-invocation"
	CodeExpansionDepthLimit     Code = "preprocess/expansion-depth-limit"
	CodeOutputSizeLimit         Code = "preprocess/output-size-limit"
	CodeMalformedInclude        Code = "preprocess/malformed-include"
	CodeIncludeNotFound         Code = "preprocess/include-not-found"
	CodeIncludeCycle            Code = "preprocess/include-cycle"
	CodeIncludeDepthLimit       Code = "preprocess/include-depth-limit"
	CodeUserError               Code = "preprocess/user-error"
	CodeUserWarning             Code = "preprocess/user-warning"
	CodeAssertFailed            Code = "preprocess/assert-failed"
	CodeAssertUnknown           Code = "preprocess/assert-unknown"
)

type CompatibleEdit

type CompatibleEdit struct {
	Before ByteRange
	After  ByteRange
}

CompatibleEdit maps one changed range between source revisions.

type Diagnostic

type Diagnostic struct {
	File     uint32
	Code     Code
	Severity diagnostic.Severity
	Message  string
	Range    ByteRange
}

Diagnostic is a preprocessor-stage finding. File indexes into Result.Files; convert to pawnkit-core's shared format with [ToCore].

func (Diagnostic) ToCore

func (d Diagnostic) ToCore(file source.FileID) diagnostic.Diagnostic

ToCore converts d into the shared diagnostic.Diagnostic interchange format. file is the caller's pawnkit-core FileID for d.File; callers processing multiple files (via #include splicing) look up Result.Files[d.File] in their own source.Registry to obtain it.

type DirectiveKind

type DirectiveKind uint8

DirectiveKind identifies which conditional-compilation directive opened a Branch.

const (
	DirectiveIf DirectiveKind = iota + 1
	DirectiveElseif
	DirectiveElse
)

func (DirectiveKind) String

func (k DirectiveKind) String() string

type FileInfo

type FileInfo struct {
	URI     string
	Depth   int
	Content []byte
	Tokens  []token.Token
}

FileInfo describes one file (root or spliced #include) contributing tokens to a Result.

type Include

type Include struct {
	File          uint32
	Path          string
	Angle         bool // <path> vs "path"
	Optional      bool // #tryinclude
	DirectiveSpan ByteRange
	Active        bool
	Resolved      bool
	ResolvedURI   string
	ChildFile     uint32
	HasChildFile  bool
}

Include records one #include/#tryinclude directive and its resolution outcome.

type IncludeResolver

type IncludeResolver interface {
	// Resolve looks up path as referenced from the file identified by
	// fromURI. angle reports whether the directive used <path> (system
	// search) rather than "path" (quoted, relative-first search).
	//
	// ok is false when the target could not be found; Run treats that as an
	// error for #include and as a silent no-op for #tryinclude.
	Resolve(fromURI, path string, angle bool) (content []byte, resolvedURI string, ok bool)
}

IncludeResolver resolves #include/#tryinclude targets to content. A host tool supplies an implementation backed by pawn-project's include search (or an in-memory map for tests); this package performs no filesystem access itself, per the architecture's narrow-interface boundary rule.

type Macro

type Macro struct {
	Name            string
	Kind            MacroKind
	ParamCount      int
	ParamSlots      map[int]int
	NamedParams     map[string]int
	FlexiblePattern bool
	Body            []ptok
	File            uint32
	DefSpan         ByteRange
}

Macro is one active #define.

func (Macro) ReplacementCallable

func (m Macro) ReplacementCallable() (string, bool)

ReplacementCallable returns the function called by a forwarding macro.

type MacroInvocation

type MacroInvocation struct {
	File  uint32
	Range ByteRange
}

MacroInvocation records a source range expanded as a macro.

type MacroKind

type MacroKind uint8

MacroKind distinguishes object-like from function-like macros.

const (
	MacroObjectLike MacroKind = iota + 1
	MacroFunctionLike
)

func (MacroKind) String

func (k MacroKind) String() string

type MapResolver

type MapResolver map[string][]byte

MapResolver is a trivial IncludeResolver backed by an in-memory map keyed by the exact path text used in the directive, ignoring fromURI and angle. Useful for tests and small embedded-include scenarios.

func (MapResolver) Resolve

func (m MapResolver) Resolve(_, path string, _ bool) ([]byte, string, bool)

type Options

type Options struct {
	// URI identifies the root file for include resolution and diagnostics.
	URI string
	// Resolver resolves #include/#tryinclude targets. A nil Resolver means
	// includes are recorded but never expanded (Include.Resolved stays
	// false), which is the honest answer when no project context is
	// available rather than guessing at file contents.
	Resolver IncludeResolver
	// Predefined seeds the macro table before processing begins (e.g. a
	// profile's built-in defines such as OPEN_MP). Values are parsed as a
	// single macro-body token run; an empty value defines an empty macro.
	Predefined map[string]string
	TokenCache *TokenCache

	MaxExpansionDepth   int
	MaxConditionalDepth int
	MaxIncludeDepth     int
	MaxOutputTokens     int
}

Options controls one preprocessing run. The zero value is usable; all limits fall back to conservative, documented defaults.

type Result

type Result struct {
	Files            []FileInfo // Files[0] is the root file.
	Source           []byte
	ExpandedSource   []byte
	OriginalTokens   []token.Token
	ExpandedTokens   []token.Token
	Branches         []Branch
	Includes         []Include
	MacroInvocations []MacroInvocation
	Macros           map[string]Macro
	Diagnostics      []Diagnostic
	Truncated        bool
}

Result is the immutable outcome of one Run. All slices are safe to retain; nothing here is mutated after Run returns.

Source and ExpandedSource are deliberately separate buffers: Source is the root file's exact bytes (what OriginalTokens and Branches/Includes with File == 0 index into), while ExpandedSource is a synthesized buffer holding the spelled text of every expanded token in emission order. A single expanded token's spelling may come from the macro-definition site, a call-site argument, or a spliced #include file - three different original buffers - so ExpandedTokens cannot index into Source directly; use each token's Origin chain (via github.com/pawnkit/pawn-parser's SyntaxToken.Origin) to recover the true original location instead.

func ReuseTriviaContext

func ReuseTriviaContext(
	ctx context.Context,
	src []byte,
	uri string,
	cache *TokenCache,
	previous *Result,
) (*Result, bool, error)

ReuseTriviaContext reuses preprocessing when only trivia text changed.

func Run

func Run(src []byte, opts Options) *Result

Run preprocesses src and returns the resulting three-view Result. Run never panics on malformed input; unbalanced or truncated constructs are reported as diagnostics and bounded by Options' limits.

func RunContext

func RunContext(ctx context.Context, src []byte, opts Options) (*Result, error)

RunContext preprocesses src and stops when ctx is cancelled.

func (*Result) ToCoreDiagnostics

func (r *Result) ToCoreDiagnostics(root source.FileID) []diagnostic.Diagnostic

ToCoreDiagnostics maps all diagnostics to one file. Use ToRegistryDiagnostics when include locations matter.

func (*Result) ToRegistryDiagnostics

func (r *Result) ToRegistryDiagnostics(registry *source.Registry, fallback source.FileID) []diagnostic.Diagnostic

ToRegistryDiagnostics maps diagnostics through Result.Files.

type TokenCache

type TokenCache struct {
	// contains filtered or unexported fields
}

func NewTokenCache

func NewTokenCache() *TokenCache