engine

Documentation

Overview

Package engine provides the SQL model execution engine. discovery.go contains the unified discovery system for macros and models.

Package engine provides the SQL model execution engine. It handles dependency resolution, topological execution, and incremental builds.

Package engine provides the SQL orchestration layer. This file contains the FormatSQL function which combines parsing and formatting.

Index

• func FormatSQL(sql string, d *core.Dialect) (string, error)
• func NewLineageExtractor() loader.LineageExtractor
• type Config
• type DiscoveryError
• type DiscoveryOptions
• type DiscoveryResult
  • func (r *DiscoveryResult) HasErrors() bool
  • func (r *DiscoveryResult) Summary() string
• type Engine
  • func New(cfg Config) (*Engine, error)
  • func (e *Engine) Close() error
  • func (e *Engine) Discover(opts DiscoveryOptions) (*DiscoveryResult, error)
  • func (e *Engine) DiscoverLegacy() error
  • func (e *Engine) EnsureConnected(ctx context.Context) error
  • func (e *Engine) GetAdapter() adapter.Adapter
  • func (e *Engine) GetDialect() *core.Dialect
  • func (e *Engine) GetGraph() *dag.Graph
  • func (e *Engine) GetModels() map[string]*core.Model
  • func (e *Engine) GetStatePath() string
  • func (e *Engine) GetStateStore() core.Store
  • func (e *Engine) LoadSeeds(ctx context.Context) error
  • func (e *Engine) RenderModel(modelPath string) (string, error)
  • func (e *Engine) RenderModelTimed(modelPath string) RenderResult
  • func (e *Engine) Run(ctx context.Context, env string) (*core.Run, error)
  • func (e *Engine) RunSelected(ctx context.Context, env string, modelPaths []string, includeDownstream bool) (*core.Run, error)
  • func (e *Engine) SetRunObserver(observer RunObserver)
• type RenderResult
• type RunObserver

Functions

func FormatSQL

func FormatSQL(sql string, d *core.Dialect) (string, error)

FormatSQL parses and formats SQL in one step. This is an orchestration function that combines parser and formatter.

func NewLineageExtractor

func NewLineageExtractor() loader.LineageExtractor

NewLineageExtractor creates a new LineageExtractor that uses the internal/lineage package.

Types

type Config

type Config struct {
	// ModelsDir is the path to the models directory
	ModelsDir string
	// SeedsDir is the path to the seeds (raw data) directory
	SeedsDir string
	// MacrosDir is the path to the macros directory (optional)
	MacrosDir string
	// StatePath is the path to the SQLite state database
	StatePath string
	// Environment is the current environment (dev, staging, prod)
	Environment string
	// Target contains adapter/database configuration
	Target *starctx.TargetInfo
	// AdapterConfig contains the full adapter configuration
	AdapterConfig *core.AdapterConfig
	// Logger is the structured logger (optional, uses discard if nil)
	Logger *slog.Logger

	// DatabasePath is the path to the DuckDB database (empty for in-memory).
	//
	// Deprecated: Use Target configuration instead.
	DatabasePath string
}

Config holds engine configuration.

type DiscoveryError

type DiscoveryError struct {
	Path    string
	Type    string // "parse", "validation", "hash", "save"
	Message string
}

DiscoveryError represents a non-fatal error during discovery.

type DiscoveryOptions

type DiscoveryOptions struct {
	ForceFullRefresh bool   // Ignore content hashes, re-parse everything
	ModelsDir        string // Override default models directory
	MacrosDir        string // Override default macros directory
	SeedsDir         string // Override default seeds directory
}

DiscoveryOptions configures the discovery process.

type DiscoveryResult

type DiscoveryResult struct {
	// Models
	ModelsTotal   int
	ModelsChanged int
	ModelsSkipped int
	ModelsDeleted int

	// Macros
	MacrosTotal   int
	MacrosChanged int
	MacrosSkipped int
	MacrosDeleted int

	// Seeds
	SeedsValidated int
	SeedsMissing   []string

	// Errors (non-fatal)
	Errors []DiscoveryError

	// Timing
	Duration time.Duration
}

DiscoveryResult contains statistics about the discovery run.

func (*DiscoveryResult) HasErrors

func (r *DiscoveryResult) HasErrors() bool

HasErrors returns true if any errors occurred.

func (*DiscoveryResult) Summary

func (r *DiscoveryResult) Summary() string

Summary returns a human-readable summary.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine orchestrates the execution of SQL models.

func New

func New(cfg Config) (*Engine, error)

New creates a new engine with lazy database connection. The database adapter is only connected when Run() or LoadSeeds() is called.

func (*Engine) Close

func (e *Engine) Close() error

Close releases all resources.

func (*Engine) Discover

func (e *Engine) Discover(opts DiscoveryOptions) (*DiscoveryResult, error)

Discover performs unified discovery of macros and models. This is the single source of truth for project state.

func (*Engine) DiscoverLegacy

func (e *Engine) DiscoverLegacy() error

DiscoverLegacy provides backward compatibility for code that uses the old Discover() signature.

Deprecated: Use Discover(opts DiscoveryOptions) instead.

func (*Engine) EnsureConnected

func (e *Engine) EnsureConnected(ctx context.Context) error

EnsureConnected ensures the database adapter is connected. This is useful when you need to run queries without running models.

func (*Engine) GetAdapter

func (e *Engine) GetAdapter() adapter.Adapter

GetAdapter returns the database adapter. Returns nil if the database is not yet connected. Call Query() on the returned adapter to execute SQL queries.

func (*Engine) GetDialect

func (e *Engine) GetDialect() *core.Dialect

GetDialect returns the SQL dialect for the connected adapter. Returns nil if the database is not yet connected.

func (*Engine) GetGraph

func (e *Engine) GetGraph() *dag.Graph

GetGraph returns the dependency graph.

func (*Engine) GetModels

func (e *Engine) GetModels() map[string]*core.Model

GetModels returns all discovered models.

func (*Engine) GetStatePath

func (e *Engine) GetStatePath() string

GetStatePath returns the path to the state database file.

func (*Engine) GetStateStore

func (e *Engine) GetStateStore() core.Store

GetStateStore returns the state store.

func (*Engine) LoadSeeds

func (e *Engine) LoadSeeds(ctx context.Context) error

LoadSeeds loads all CSV files from the seeds directory into the database.

func (*Engine) RenderModel

func (e *Engine) RenderModel(modelPath string) (string, error)

RenderModel renders the SQL for a model with all templates expanded. This is the public API for SQL rendering.

func (*Engine) RenderModelTimed

func (e *Engine) RenderModelTimed(modelPath string) RenderResult

RenderModelTimed renders a model and returns timing information.

func (*Engine) Run

func (e *Engine) Run(ctx context.Context, env string) (*core.Run, error)

Run executes all models in topological order using a two-phase approach: Phase 1: Validate all templates (fail fast if any fail) Phase 2: Execute all models

func (*Engine) RunSelected

func (e *Engine) RunSelected(ctx context.Context, env string, modelPaths []string, includeDownstream bool) (*core.Run, error)

RunSelected executes only the specified models and their downstream dependents. Uses a two-phase approach: validate all templates, then execute. Upstream dependencies must already exist in the database.

func (*Engine) SetRunObserver

func (e *Engine) SetRunObserver(observer RunObserver)

SetRunObserver sets an observer to receive run lifecycle events. Pass nil to remove the observer. Thread-safe.

type RenderResult

type RenderResult struct {
	SQL      string
	RenderMS int64
	Error    error
}

RenderResult contains rendered SQL and timing information.

type RunObserver

type RunObserver interface {
	OnRunStarted(run *core.Run)
	OnModelRunUpdated(runID string, modelRun *core.ModelRun)
	OnRunCompleted(run *core.Run)
}

RunObserver receives notifications about run state changes. Implementations can use these callbacks to update UIs, trigger notifications, etc.