gosqlx

README

GoSQLX

⚡ High-Performance SQL Parser for Go ⚡

Production-ready, high-performance SQL parsing SDK for Go
Zero-copy tokenization • Object pooling • Multi-dialect support • Unicode-first design

🚀 Installation • ⚡ Quick Start • 📚 Documentation • 💡 Examples • 📊 Benchmarks

---

🎯 Overview

GoSQLX is a high-performance SQL parsing library designed for production use. It provides zero-copy tokenization, intelligent object pooling, and comprehensive SQL dialect support while maintaining a simple, idiomatic Go API.

✨ Key Features

• 🚀 Blazing Fast: 2.2M ops/sec, 8M tokens/sec processing speed
• 💾 Memory Efficient: 60-80% reduction through intelligent object pooling
• 🔒 Thread-Safe: Race-free, linear scaling to 128+ cores
• 🔗 Complete JOIN Support: All JOIN types (INNER/LEFT/RIGHT/FULL OUTER/CROSS/NATURAL) with proper tree logic
• 🌍 Unicode Support: Complete UTF-8 support for international SQL
• 🔧 Multi-Dialect: PostgreSQL, MySQL, SQL Server, Oracle, SQLite
• 📊 Zero-Copy: Direct byte slice operations, < 200ns latency
• 🏗️ Production Ready: Battle-tested with 0 race conditions detected

🎯 Performance Highlights (v1.1.0)

2.2M	8M	184ns	60-80%	15+
Ops/sec	Tokens/sec	Latency	Memory Saved	JOIN Tests

✅ Complete JOIN Support • Zero race conditions • Left-associative join trees • Production validated

📈 Project Stats

📦 Installation

go get github.com/ajitpratap0/GoSQLX

Requirements:

• Go 1.19 or higher
• No external dependencies

🚀 Quick Start

Basic Usage

package main

import (
    "fmt"
    "log"
    
    "github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
)

func main() {
    // Get tokenizer from pool (always return it!)
    tkz := tokenizer.GetTokenizer()
    defer tokenizer.PutTokenizer(tkz)
    
    // Tokenize SQL
    sql := "SELECT id, name FROM users WHERE age > 18"
    tokens, err := tkz.Tokenize([]byte(sql))
    if err != nil {
        log.Fatal(err)
    }
    
    // Process tokens
    fmt.Printf("Generated %d tokens\n", len(tokens))
    for _, token := range tokens {
        fmt.Printf("  %s (line %d, col %d)\n", 
            token.Token.Value,
            token.Start.Line,
            token.Start.Column)
    }
}

Advanced Example with AST

package main

import (
    "fmt"
    
    "github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
    "github.com/ajitpratap0/GoSQLX/pkg/sql/parser"
)

func AnalyzeSQL(sql string) error {
    // Tokenize
    tkz := tokenizer.GetTokenizer()
    defer tokenizer.PutTokenizer(tkz)
    
    tokens, err := tkz.Tokenize([]byte(sql))
    if err != nil {
        return fmt.Errorf("tokenization failed: %w", err)
    }
    
    // Parse to AST
    p := parser.NewParser()
    defer p.Release()
    
    ast, err := p.Parse(convertTokens(tokens))
    if err != nil {
        return fmt.Errorf("parsing failed: %w", err)
    }
    
    // Analyze AST
    fmt.Printf("Statement type: %T\n", ast)
    return nil
}

📚 Documentation

📖 Comprehensive Guides

Guide	Description
API Reference	Complete API documentation with examples
Usage Guide	Detailed patterns and best practices
Architecture	System design and internal architecture
Troubleshooting	Common issues and solutions

🚀 Getting Started

Document	Purpose
Production Guide	Deployment and monitoring
SQL Compatibility	Dialect support matrix
Security Analysis	Security assessment
Examples	Working code examples

📋 Quick Links

• Installation & Setup
• Basic Usage
• Advanced Patterns
• Performance Tuning
• Error Handling
• FAQ

🔗 JOIN Support (v1.1.0)

GoSQLX now supports all JOIN types with proper left-associative tree logic:

// Complex JOIN query with multiple table relationships
sql := `
    SELECT u.name, o.order_date, p.product_name, c.category_name
    FROM users u
    LEFT JOIN orders o ON u.id = o.user_id  
    INNER JOIN products p ON o.product_id = p.id
    RIGHT JOIN categories c ON p.category_id = c.id
    WHERE u.active = true
    ORDER BY o.order_date DESC
`

// Parse with automatic JOIN tree construction
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)

tokens, err := tkz.Tokenize([]byte(sql))
parser := parser.NewParser()
ast, err := parser.Parse(tokens)

// Access JOIN information
if selectStmt, ok := ast.Statements[0].(*ast.SelectStatement); ok {
    fmt.Printf("Found %d JOINs:\n", len(selectStmt.Joins))
    for i, join := range selectStmt.Joins {
        fmt.Printf("JOIN %d: %s (left: %s, right: %s)\n", 
            i+1, join.Type, join.Left.Name, join.Right.Name)
    }
}

Supported JOIN Types:

• ✅ INNER JOIN - Standard inner joins
• ✅ LEFT JOIN / LEFT OUTER JOIN - Left outer joins
• ✅ RIGHT JOIN / RIGHT OUTER JOIN - Right outer joins
• ✅ FULL JOIN / FULL OUTER JOIN - Full outer joins
• ✅ CROSS JOIN - Cartesian product joins
• ✅ NATURAL JOIN - Natural joins (implicit ON clause)
• ✅ USING (column) - Single-column using clause

💻 Examples

Multi-Dialect Support

// PostgreSQL with array operators
sql := `SELECT * FROM users WHERE tags @> ARRAY['admin']`

// MySQL with backticks
sql := "SELECT `user_id`, `name` FROM `users`"

// SQL Server with brackets
sql := "SELECT [user_id], [name] FROM [users]"

Unicode and International SQL

// Japanese
sql := `SELECT "名前", "年齢" FROM "ユーザー"`

// Russian
sql := `SELECT "имя", "возраст" FROM "пользователи"`

// Arabic
sql := `SELECT "الاسم", "العمر" FROM "المستخدمون"`

// Emoji support
sql := `SELECT * FROM users WHERE status = '🚀'`

Concurrent Processing

func ProcessConcurrently(queries []string) {
    var wg sync.WaitGroup
    
    for _, sql := range queries {
        wg.Add(1)
        go func(query string) {
            defer wg.Done()
            
            // Each goroutine gets its own tokenizer
            tkz := tokenizer.GetTokenizer()
            defer tokenizer.PutTokenizer(tkz)
            
            tokens, _ := tkz.Tokenize([]byte(query))
            // Process tokens...
        }(sql)
    }
    
    wg.Wait()
}

📊 Performance

🎯 v1.0.0 Performance Improvements

Metric	Previous	v1.0.0	Improvement
Throughput	1.5M ops/s	2.2M ops/s	+47% ✅
Token Processing	5M tokens/s	8M tokens/s	+60% ✅
Concurrency	Limited	Linear to 128 cores	∞ ✅
Memory Usage	Baseline	60-80% reduction	-70% ✅
Latency (p99)	1μs	184ns	-82% ✅

Latest Benchmark Results

BenchmarkTokenizer/SimpleSQL-16             965,466      1,238 ns/op     1,585 B/op      20 allocs/op
BenchmarkTokenizer/ComplexSQL-16             92,636     13,078 ns/op    13,868 B/op     159 allocs/op
BenchmarkTokenizer/Concurrent-128-16        639,093      1,788 ns/op    10,735 B/op      88 allocs/op

BenchmarkParser/SimpleSelect-16           6,330,259        185 ns/op       536 B/op       9 allocs/op
BenchmarkParser/ParallelSelect-16         8,175,652        154 ns/op       536 B/op       9 allocs/op

BenchmarkThroughput/200_goroutines-16     3,144,678        381 ns/op   2,189,740 ops/sec
BenchmarkTokensPerSecond-16                 733,141      1,619 ns/op   8,032,114 tokens/sec

Performance Characteristics

Metric	Value	Details
Throughput	2.2M ops/sec	200 concurrent goroutines
Token Rate	8M tokens/sec	Sustained processing
Latency	< 200ns	Simple queries (p50)
Memory	1.6KB/query	Simple SQL with pooling
Scaling	Linear to 128	Perfect concurrency
Pool Efficiency	95%+ hit rate	Effective reuse

See PERFORMANCE_REPORT.md for detailed analysis.

🧪 Testing

# Run all tests with race detection
go test -race ./...

# Run benchmarks
go test -bench=. -benchmem ./...

# Generate coverage report
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

# Run specific test suites
go test -v ./pkg/sql/tokenizer/
go test -v ./pkg/sql/parser/

🏗️ Project Structure

GoSQLX/
├── pkg/
│   ├── models/              # Core data structures
│   │   ├── token.go        # Token definitions
│   │   └── location.go     # Position tracking
│   └── sql/
│       ├── tokenizer/       # Lexical analysis
│       │   ├── tokenizer.go
│       │   └── pool.go
│       ├── parser/          # Syntax analysis
│       │   ├── parser.go
│       │   └── expressions.go
│       ├── ast/            # Abstract syntax tree
│       │   ├── nodes.go
│       │   └── statements.go
│       └── keywords/        # SQL keywords
├── examples/               # Usage examples
│   └── cmd/
│       ├── example.go
│       └── example_test.go
├── docs/                   # Documentation
│   ├── API_REFERENCE.md
│   ├── USAGE_GUIDE.md
│   ├── ARCHITECTURE.md
│   └── TROUBLESHOOTING.md
└── tools/                  # Development tools

🛠️ Development

Prerequisites

• Go 1.19+
• Make (optional, for Makefile targets)
• golint, staticcheck (for code quality)

Building

# Build the project
make build

# Run quality checks
make quality

# Run all tests
make test

# Clean build artifacts
make clean

Code Quality

# Format code
go fmt ./...

# Vet code
go vet ./...

# Run linter
golint ./...

# Static analysis
staticcheck ./...

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

How to Contribute

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Development Guidelines

• Write tests for new features
• Ensure all tests pass with race detection
• Follow Go idioms and best practices
• Update documentation for API changes
• Add benchmarks for performance-critical code

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🚀 Roadmap

Phase 1: Core SQL Enhancements (Q1 2025) - v1.1.0 ✅

• ✅ Complete JOIN support (INNER/LEFT/RIGHT/FULL OUTER/CROSS/NATURAL)
• ✅ Proper join tree logic with left-associative relationships
• ✅ USING clause parsing (single-column, multi-column planned for Phase 2)
• ✅ Enhanced error handling with contextual JOIN error messages
• ✅ Comprehensive test coverage (15+ JOIN scenarios including error cases)
• 🏗️ CTE foundation laid (AST structures, tokens, parser integration points)

Phase 2: CTE & Advanced Features (Q2 2025) - v1.2.0

• 📋 Common Table Expressions (CTEs) with RECURSIVE support
• 📋 Set operations (UNION/EXCEPT/INTERSECT with ALL modifier)
• 📋 Multi-column USING clause support
• 📋 Window functions (OVER, PARTITION BY)
• 📋 Advanced subqueries in all contexts

Phase 3: Dialect Specialization (Q1 2025) - v2.0.0

• 📋 PostgreSQL arrays, JSONB, custom types
• 📋 MySQL-specific syntax and functions
• 📋 SQL Server T-SQL extensions
• 📋 Multi-dialect parser with auto-detection

Phase 4: Intelligence Layer (Q2 2025) - v2.1.0

• 📋 Query optimization suggestions
• 📋 Security vulnerability detection
• 📋 Performance analysis and hints
• 📋 Schema validation

📄 Full Architectural Review & Roadmap

🤝 Community & Support

Join Our Community

Get Help

Channel	Purpose	Response Time
🐛 Bug Reports	Report issues	Community-driven
💡 Feature Requests	Suggest improvements	Community-driven
💬 Discussions	Q&A, ideas, showcase	Community-driven
🔒 Security	Report vulnerabilities	Best effort

👥 Contributors

Core Team

How to Contribute

We love your input! We want to make contributing as easy and transparent as possible.

Quick Contribution Guide

1. 🍴 Fork the repo
2. 🔨 Make your changes
3. ✅ Ensure tests pass (go test -race ./...)
4. 📝 Update documentation
5. 🚀 Submit a PR

🎯 Use Cases

Industry	Use Case	Benefits
🏦 FinTech	SQL validation & auditing	Fast validation, compliance tracking
📊 Analytics	Query parsing & optimization	Real-time analysis, performance insights
🛡️ Security	SQL injection detection	Pattern matching, threat prevention
🏗️ DevTools	IDE integration & linting	Syntax highlighting, auto-completion
📚 Education	SQL learning platforms	Interactive parsing, error explanation
🔄 Migration	Cross-database migration	Dialect conversion, compatibility check

📊 Who's Using GoSQLX

Using GoSQLX in production? Let us know!

📈 Project Metrics

Performance Benchmarks

graph LR
    A[SQL Input] -->|2.2M ops/sec| B[Tokenizer]
    B -->|8M tokens/sec| C[Parser]
    C -->|Zero-copy| D[AST]
    D -->|60-80% less memory| E[Output]

🗺️ Roadmap

Release Timeline

Version	Status	Release Date	Features
v0.9.0	✅ Released	2024-01-15	Initial release
v1.0.0	✅ Released	2024-12-01	Production ready, +47% performance
v1.1.0	🎉 Current	2025-01-03	Complete JOIN support, error handling
v1.2.0	📝 Planned	Q2 2025	CTEs, set operations, multi-column USING
v2.0.0	🔮 Future	Q4 2025	Dialect specialization, advanced features

💖 Support This Project

If GoSQLX helps your project, please consider:

Other Ways to Support

• ⭐ Star this repository
• 🐦 Tweet about GoSQLX
• 📝 Write a blog post
• 🎥 Create a tutorial
• 🐛 Report bugs
• 💡 Suggest features
• 🔧 Submit PRs

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Built with ❤️ by the GoSQLX Team

_{Copyright © 2024 GoSQLX. All rights reserved.}

Documentation

Overview

Package gosqlx provides a high-performance SQL parsing SDK for Go with zero-copy tokenization and object pooling. It offers production-ready SQL lexing, parsing, and AST generation with support for multiple SQL dialects and advanced SQL features.

Core Features:

- Zero-copy tokenization for optimal performance - Object pooling for 60-80% memory reduction - Multi-dialect SQL support (PostgreSQL, MySQL, SQL Server, Oracle, SQLite) - Thread-safe implementation with linear scaling to 128+ cores - Full Unicode/UTF-8 support for international SQL - Performance monitoring and metrics collection - Visitor pattern support for AST traversal

Advanced SQL Features (Phase 2 - v1.2.0+):

- Common Table Expressions (CTEs) with WITH clause - Recursive CTEs with WITH RECURSIVE support - Multiple CTEs in single query - Set operations: UNION, UNION ALL, EXCEPT, INTERSECT - Complex query compositions and left-associative parsing - ~70% SQL-92 standards compliance

Basic Usage:

import (
    "github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer"
    "github.com/ajitpratap0/GoSQLX/pkg/sql/parser"
    "github.com/ajitpratap0/GoSQLX/pkg/sql/ast"
)

// Get a tokenizer from the pool
tkz := tokenizer.GetTokenizer()
defer tokenizer.PutTokenizer(tkz)

// Tokenize SQL
tokens, err := tkz.Tokenize([]byte("SELECT * FROM users WHERE id = 1"))
if err != nil {
    log.Fatal(err)
}

// Parse tokens into AST
p := &parser.Parser{}
astObj, err := p.Parse(tokens)
if err != nil {
    log.Fatal(err)
}
defer ast.ReleaseAST(astObj)

Advanced Usage (Phase 2 Features):

// Common Table Expression (CTE)
cteSQL := `WITH sales_summary AS (
    SELECT region, SUM(amount) as total
    FROM sales
    GROUP BY region
) SELECT region FROM sales_summary WHERE total > 1000`

// Recursive CTE
recursiveSQL := `WITH RECURSIVE employee_tree AS (
    SELECT employee_id, manager_id, name FROM employees WHERE manager_id IS NULL
    UNION ALL
    SELECT e.employee_id, e.manager_id, e.name
    FROM employees e JOIN employee_tree et ON e.manager_id = et.employee_id
) SELECT * FROM employee_tree`

// Set Operations
unionSQL := `SELECT name FROM customers UNION SELECT name FROM suppliers`
exceptSQL := `SELECT product FROM inventory EXCEPT SELECT product FROM discontinued`
intersectSQL := `SELECT customer_id FROM orders INTERSECT SELECT customer_id FROM payments`

Performance:

GoSQLX achieves: - 946K+ sustained operations/second (30s load testing) - 1.25M+ operations/second peak throughput (concurrent) - 8M+ tokens/second processing speed - <280ns latency for simple queries - <1μs latency for complex queries with CTEs/set operations - Linear scaling to 128+ cores - 60-80% memory reduction with object pooling - Zero memory leaks under extended load - Race-free concurrent operation validated

For more examples and detailed documentation, see: https://github.com/ajitpratap0/GoSQLX