README
¶
LeapSQL
A SQL transformation framework that automatically detects dependencies from your queries. Write pure SQL, get dependency graphs, column-level lineage, and incremental builds - no ref() functions or boilerplate required.
Why LeapSQL?
- Automatic Dependency Detection - Write standard SQL. LeapSQL parses your queries to build the DAG automatically from
FROM,JOIN, subqueries, and CTEs. - Column-Level Lineage - Track exactly how each field flows through your transformations, not just table-level dependencies.
- Starlark Templating - Use a Python-like language for safe, deterministic SQL generation. No more Jinja recursion errors.
- Single Binary - Distributed as a standalone Go binary. No Python environments or dependency management.
- Database-Backed State - Run history and metadata stored in SQLite, not scattered JSON artifacts.
Quick Example
/*---
name: customers
materialized: table
---*/
SELECT
c.id,
c.name,
c.email,
COUNT(o.id) as order_count,
SUM(o.amount) as total_spent
FROM raw_customers c
LEFT JOIN raw_orders o ON c.id = o.customer_id
GROUP BY c.id, c.name, c.email
LeapSQL automatically detects that this model depends on raw_customers and raw_orders, tracks that order_count derives from orders.id, and ensures models build in the correct order.
Installation
From Source
Requires Go 1.21+:
go install github.com/leapstack-labs/leapsql/cmd/leapsql@latest
Pre-built Binaries
Download from the Releases Page, or:
# macOS / Linux
curl -L https://github.com/leapstack-labs/leapsql/releases/latest/download/leapsql-$(uname -s)-$(uname -m) -o leapsql
chmod +x leapsql
sudo mv leapsql /usr/local/bin/
Quickstart
# Create a new project
mkdir my-project && cd my-project
leapsql init --example
# Load seed data
leapsql seed
# View the dependency graph
leapsql dag
# Run all models
leapsql run
CLI Commands
| Command | Description |
|---|---|
init |
Initialize a new project |
run |
Execute models in dependency order |
list |
List all discovered models |
dag |
Display the dependency graph |
lineage |
Show upstream/downstream dependencies for a model |
render |
Output compiled SQL with templates expanded |
seed |
Load CSV files into the database |
docs |
Generate or serve documentation |
lsp |
Start the language server for IDE integration |
Run leapsql <command> --help for detailed usage.
Supported Databases
| Database | Status | Notes |
|---|---|---|
| DuckDB | Stable | Embedded, great for local development |
| PostgreSQL | Stable | Production-ready with pgx driver |
Configure your adapter in leapsql.yaml:
adapter:
type: duckdb
path: ./warehouse.db
IDE Support
LeapSQL includes a Language Server Protocol (LSP) implementation providing:
- Autocomplete for models and macros
- Hover documentation
- Go-to-definition
A VS Code extension is available in the vscode-leapsql/ directory.
Documentation
Full documentation: leapstack-labs.github.io/leapsql
Contributing
Contributions are welcome. To get started:
# Run tests
task test
# Run linter
task lint
# Run both
task check
See the Taskfile.yml for all available commands.
License
MIT License. See LICENSE for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
leapsql
command
Package main provides the CLI entry point for LeapSQL.
|
Package main provides the CLI entry point for LeapSQL. |
|
internal
|
|
|
cli
Package cli provides the command-line interface for LeapSQL.
|
Package cli provides the command-line interface for LeapSQL. |
|
cli/config
Package config provides configuration management for LeapSQL CLI.
|
Package config provides configuration management for LeapSQL CLI. |
|
cli/testutil
Package testutil provides test utilities for CLI testing.
|
Package testutil provides test utilities for CLI testing. |
|
config
Package config provides shared configuration types for LeapSQL.
|
Package config provides shared configuration types for LeapSQL. |
|
dag
Package dag provides directed acyclic graph operations for model dependencies.
|
Package dag provides directed acyclic graph operations for model dependencies. |
|
engine
Package engine provides the SQL model execution engine.
|
Package engine provides the SQL model execution engine. |
|
lineage
Package lineage provides domain-specific SQL lineage extraction.
|
Package lineage provides domain-specific SQL lineage extraction. |
|
loader
Package loader provides YAML frontmatter parsing for SQL model files.
|
Package loader provides YAML frontmatter parsing for SQL model files. |
|
lsp
Package lsp implements a Language Server Protocol server for LeapSQL.
|
Package lsp implements a Language Server Protocol server for LeapSQL. |
|
macro
Package macro provides functionality for loading and managing Starlark macros.
|
Package macro provides functionality for loading and managing Starlark macros. |
|
provider
Package provider manages shared context for all lint consumers.
|
Package provider manages shared context for all lint consumers. |
|
registry
Package registry provides model registration and table name resolution.
|
Package registry provides model registration and table name resolution. |
|
starlark
Package starlark provides Starlark execution context and builtins for template rendering.
|
Package starlark provides Starlark execution context and builtins for template rendering. |
|
state
Package state provides state management with database migrations.
|
Package state provides state management with database migrations. |
|
template
Package template provides a template processor for SQL files with Starlark expressions.
|
Package template provides a template processor for SQL files with Starlark expressions. |
|
testutil
Package testutil provides test utilities for structured logging.
|
Package testutil provides test utilities for structured logging. |
|
ui
Package ui provides a web-based development UI for LeapSQL.
|
Package ui provides a web-based development UI for LeapSQL. |
|
ui/features
Package features provides shared test utilities for UI feature tests.
|
Package features provides shared test utilities for UI feature tests. |
|
ui/features/common
Package common provides shared types and utilities for UI features.
|
Package common provides shared types and utilities for UI features. |
|
ui/features/common/components
templ: version: v0.3.977
|
templ: version: v0.3.977 |
|
ui/features/common/layouts
templ: version: v0.3.977
|
templ: version: v0.3.977 |
|
ui/features/database
Package database provides database browser handlers for the UI.
|
Package database provides database browser handlers for the UI. |
|
ui/features/graph
Package graph provides DAG visualization handlers for the UI.
|
Package graph provides DAG visualization handlers for the UI. |
|
ui/features/home
Package home provides the home/landing page feature for the UI.
|
Package home provides the home/landing page feature for the UI. |
|
ui/features/macros
Package macros provides the macros catalog feature for the UI.
|
Package macros provides the macros catalog feature for the UI. |
|
ui/features/models
Package models provides model detail handlers for the UI.
|
Package models provides model detail handlers for the UI. |
|
ui/features/runs
Package runs provides run history handlers for the UI.
|
Package runs provides run history handlers for the UI. |
|
ui/features/statequery
Package statequery provides handlers for querying the state database.
|
Package statequery provides handlers for querying the state database. |
|
ui/notifier
Package notifier provides a simple broadcast mechanism for SSE updates.
|
Package notifier provides a simple broadcast mechanism for SSE updates. |
|
ui/resources
Package resources provides static asset handling for the UI server.
|
Package resources provides static asset handling for the UI server. |
|
ui/router
Package router sets up HTTP routes for the UI server.
|
Package router sets up HTTP routes for the UI server. |
|
pkg
|
|
|
adapter
Package adapter provides database adapter interfaces and implementations for LeapSQL's data transformation engine.
|
Package adapter provides database adapter interfaces and implementations for LeapSQL's data transformation engine. |
|
adapters/databricks
Package databricks provides the Databricks SQL adapter for LeapSQL.
|
Package databricks provides the Databricks SQL adapter for LeapSQL. |
|
adapters/duckdb
Package duckdb provides a DuckDB database adapter for LeapSQL.
|
Package duckdb provides a DuckDB database adapter for LeapSQL. |
|
adapters/postgres
Package postgres provides a PostgreSQL database adapter for LeapSQL.
|
Package postgres provides a PostgreSQL database adapter for LeapSQL. |
|
core
Package core defines the shared language of the LeapSQL system.
|
Package core defines the shared language of the LeapSQL system. |
|
dialect
Package dialect provides SQL dialect configuration and function classification.
|
Package dialect provides SQL dialect configuration and function classification. |
|
dialects/databricks
Package databricks provides the Databricks SQL dialect definition.
|
Package databricks provides the Databricks SQL dialect definition. |
|
dialects/duckdb
Package duckdb provides the DuckDB SQL dialect definition.
|
Package duckdb provides the DuckDB SQL dialect definition. |
|
dialects/postgres
Package postgres provides the PostgreSQL SQL dialect definition.
|
Package postgres provides the PostgreSQL SQL dialect definition. |
|
dialects/snowflake
Package snowflake provides the Snowflake SQL dialect definition.
|
Package snowflake provides the Snowflake SQL dialect definition. |
|
format
Package format provides SQL statement formatting.
|
Package format provides SQL statement formatting. |
|
lint
Package lint provides a unified SQL and project linting framework.
|
Package lint provides a unified SQL and project linting framework. |
|
lint/project
Package project provides project-level linting for LeapSQL.
|
Package project provides project-level linting for LeapSQL. |
|
lint/project/rules
Package projectrules registers all project health lint rules.
|
Package projectrules registers all project health lint rules. |
|
lint/sql
Package sql provides SQL statement-level linting rules and analysis.
|
Package sql provides SQL statement-level linting rules and analysis. |
|
lint/sql/internal/ast
Package ast provides AST traversal utilities for SQL lint rules.
|
Package ast provides AST traversal utilities for SQL lint rules. |
|
lint/sql/rules
Package rules contains all SQL lint rules.
|
Package rules contains all SQL lint rules. |
|
parser
Package parser provides SQL parsing with dialect-aware syntax validation.
|
Package parser provides SQL parsing with dialect-aware syntax validation. |
|
spi
Package spi provides Service Provider Interface types for dialect clause handlers to interact with the parser without circular dependencies.
|
Package spi provides Service Provider Interface types for dialect clause handlers to interact with the parser without circular dependencies. |
|
token
Package token defines the token types for SQL parsing.
|
Package token defines the token types for SQL parsing. |
|
scripts
|
|
|
gendatabricks
command
Package main provides a web scraper that extracts Databricks SQL function metadata from the Databricks documentation and generates Go code for the dialect package.
|
Package main provides a web scraper that extracts Databricks SQL function metadata from the Databricks documentation and generates Go code for the dialect package. |
|
gendialect
command
Package main provides a generator that extracts DuckDB function metadata and generates Go code for the dialect package.
|
Package main provides a generator that extracts DuckDB function metadata and generates Go code for the dialect package. |
|
gendocs
command
Package main provides a generator that extracts CLI, schema, globals, and lint metadata from LeapSQL source code and generates markdown documentation.
|
Package main provides a generator that extracts CLI, schema, globals, and lint metadata from LeapSQL source code and generates markdown documentation. |
|
gensnowflake
command
Package main provides a web scraper that extracts Snowflake SQL function metadata from the Snowflake documentation and generates Go code for the dialect package.
|
Package main provides a web scraper that extracts Snowflake SQL function metadata from the Snowflake documentation and generates Go code for the dialect package. |
|
syncdatastardocs
command
Package main scrapes data-star.dev documentation and saves as markdown files.
|
Package main scrapes data-star.dev documentation and saves as markdown files. |
Click to show internal directories.
Click to hide internal directories.