parser

package module
v0.0.0-...-49b7217 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 8, 2025 License: Apache-2.0 Imports: 27 Imported by: 1

README

MySQL Parser

A MySQL-compatible SQL parser extracted from TiDB, now available as a standalone Go module with WASM support.

Worth mentioning the database built on top of mysql-parser: MIST

Features

  • MySQL Compatibility: Supports MySQL 5.7+ SQL syntax
  • Full AST Support: Complete Abstract Syntax Tree generation
  • Visitor Pattern: Easy AST traversal and manipulation
  • WASM Support: Runs in WebAssembly environments
  • Type Safety: Proper handling of MySQL data types
  • Cross-Platform: Optimized for both native and WASM execution

Module Information

Module Path: github.com/abbychau/mysql-parser

This parser provides MySQL-compatible SQL parsing with full AST support and two parser driver options:

  1. Test Driver (test_driver) - Lightweight, basic functionality
  2. Full Driver (parser_driver) - Complete functionality with WASM compatibility

Installation

go get github.com/abbychau/mysql-parser

Usage Examples

Basic Usage with Test Driver
package main

import (
    "fmt"
    "github.com/abbychau/mysql-parser"
    "github.com/abbychau/mysql-parser/ast"
    _ "github.com/abbychau/mysql-parser/test_driver"
)

func main() {
    p := parser.New()
    sql := "SELECT name, age FROM users WHERE age > 25"
    
    stmtNodes, _, err := p.ParseSQL(sql)
    if err != nil {
        fmt.Printf("Parse error: %v\n", err)
        return
    }
    
    fmt.Printf("Successfully parsed %d statements\n", len(stmtNodes))
}
Advanced Usage with Full Parser Driver
package main

import (
    "fmt"
    "github.com/abbychau/mysql-parser"
    "github.com/abbychau/mysql-parser/ast"
    _ "github.com/abbychau/mysql-parser/parser_driver"
)

// ColumnExtractor implements ast.Visitor to extract column names
type ColumnExtractor struct {
    ColNames []string
}

func (v *ColumnExtractor) Enter(in ast.Node) (ast.Node, bool) {
    if name, ok := in.(*ast.ColumnName); ok {
        v.ColNames = append(v.ColNames, name.Name.O)
    }
    return in, false
}

func (v *ColumnExtractor) Leave(in ast.Node) (ast.Node, bool) {
    return in, true
}

func main() {
    p := parser.New()
    sql := "SELECT users.name, orders.total FROM users JOIN orders ON users.id = orders.user_id"
    
    stmtNodes, _, err := p.ParseSQL(sql)
    if err != nil {
        fmt.Printf("Parse error: %v\n", err)
        return
    }

    // Extract column names using visitor pattern
    extractor := &ColumnExtractor{}
    stmtNodes[0].Accept(extractor)
    
    fmt.Printf("Extracted columns: %v\n", extractor.ColNames)
}
WASM Usage

For WebAssembly applications, use the full parser driver:

package main

import (
    "syscall/js"
    "github.com/abbychau/mysql-parser"
    "github.com/abbychau/mysql-parser/ast"
    _ "github.com/abbychau/mysql-parser/parser_driver"
)

func parseSQL(this js.Value, args []js.Value) interface{} {
    sqlText := args[0].String()
    
    p := parser.New()
    stmtNodes, _, err := p.ParseSQL(sqlText)
    if err != nil {
        return map[string]interface{}{
            "success": false,
            "error":   err.Error(),
        }
    }
    
    return map[string]interface{}{
        "success": true,
        "count":   len(stmtNodes),
    }
}

func main() {
    js.Global().Set("parseSQL", js.FuncOf(parseSQL))
    select {} // Keep the program running
}

Project Structure

github.com/abbychau/mysql-parser/
├── ast/                    # Abstract Syntax Tree definitions
├── charset/               # Character set support
├── mysql/                 # MySQL constants and utilities
├── parser_driver/         # Full parser driver (WASM-compatible)
├── test_driver/          # Lightweight test driver
├── wasm-demo/            # WebAssembly demo application
└── docs/                 # Documentation

Parser Drivers

Test Driver (test_driver)
  • Lightweight and minimal dependencies
  • Basic AST node support
  • Suitable for simple parsing tasks
  • Import: _ "github.com/abbychau/mysql-parser/test_driver"
Full Parser Driver (parser_driver)
  • Complete functionality with proper type handling
  • WASM-compatible (no CGO or unsafe operations)
  • Supports all MySQL data types and expressions
  • Full type system including Datum, FieldType, MyDecimal, Time
  • Comprehensive error handling and type conversions
  • Self-contained with no external dependencies
  • Import: _ "github.com/abbychau/mysql-parser/parser_driver"

Building for WASM

cd wasm-demo
GOOS=js GOARCH=wasm go build -o parser.wasm main.go

The parser_driver package is specifically designed to work in WASM environments. The demo includes a web interface for testing SQL parsing in the browser.

License

Apache License 2.0 - see LICENSE file for details.

Original Source

This parser is based on TiDB's SQL parser component, modified for standalone use and WASM compatibility.

Documentation

Overview

Package parser_driver_init initializes the parser_driver for the main parser

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrWarnOptimizerHintUnsupportedHint = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintUnsupportedHint)
	ErrWarnOptimizerHintInvalidToken    = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintInvalidToken)
	ErrWarnMemoryQuotaOverflow          = terror.ClassParser.NewStd(mysql.ErrWarnMemoryQuotaOverflow)
	ErrWarnOptimizerHintParseError      = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintParseError)
	ErrWarnOptimizerHintInvalidInteger  = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintInvalidInteger)
	ErrWarnOptimizerHintWrongPos        = terror.ClassParser.NewStd(mysql.ErrWarnOptimizerHintWrongPos)
)
View Source 
var (
	// ErrSyntax returns for sql syntax error.
	ErrSyntax = terror.ClassParser.NewStd(mysql.ErrSyntax)
	// ErrParse returns for sql parse error.
	ErrParse = terror.ClassParser.NewStd(mysql.ErrParse)
	// ErrUnknownCharacterSet returns for no character set found error.
	ErrUnknownCharacterSet = terror.ClassParser.NewStd(mysql.ErrUnknownCharacterSet)
	// ErrInvalidYearColumnLength returns for illegal column length for year type.
	ErrInvalidYearColumnLength = terror.ClassParser.NewStd(mysql.ErrInvalidYearColumnLength)
	// ErrWrongArguments returns for illegal argument.
	ErrWrongArguments = terror.ClassParser.NewStd(mysql.ErrWrongArguments)
	// ErrWrongFieldTerminators returns for illegal field terminators.
	ErrWrongFieldTerminators = terror.ClassParser.NewStd(mysql.ErrWrongFieldTerminators)
	// ErrTooBigDisplayWidth returns for data display width exceed limit .
	ErrTooBigDisplayWidth = terror.ClassParser.NewStd(mysql.ErrTooBigDisplaywidth)
	// ErrTooBigPrecision returns for data precision exceed limit.
	ErrTooBigPrecision = terror.ClassParser.NewStd(mysql.ErrTooBigPrecision)
	// ErrUnknownAlterLock returns for no alter lock type found error.
	ErrUnknownAlterLock = terror.ClassParser.NewStd(mysql.ErrUnknownAlterLock)
	// ErrUnknownAlterAlgorithm returns for no alter algorithm found error.
	ErrUnknownAlterAlgorithm = terror.ClassParser.NewStd(mysql.ErrUnknownAlterAlgorithm)
	// ErrWrongValue returns for wrong value
	ErrWrongValue = terror.ClassParser.NewStd(mysql.ErrWrongValue)
	// ErrWarnDeprecatedSyntax return when the syntax was deprecated
	ErrWarnDeprecatedSyntax = terror.ClassParser.NewStd(mysql.ErrWarnDeprecatedSyntax)
	// ErrWarnDeprecatedSyntaxNoReplacement return when the syntax was deprecated and there is no replacement.
	ErrWarnDeprecatedSyntaxNoReplacement = terror.ClassParser.NewStd(mysql.ErrWarnDeprecatedSyntaxNoReplacement)
	// ErrWrongUsage returns for incorrect usages.
	ErrWrongUsage = terror.ClassParser.NewStd(mysql.ErrWrongUsage)
	// ErrWrongDBName returns for incorrect DB name.
	ErrWrongDBName = terror.ClassParser.NewStd(mysql.ErrWrongDBName)
	// SpecFieldPattern special result field pattern
	SpecFieldPattern = regexp.MustCompile(`(\/\*!(M?[0-9]{5,6})?|\*\/)`)
)
View Source 
var Keywords = []KeywordsType{}/* 665 elements not displayed */

Keywords is used for all keywords in TiDB

Functions

func Normalize 

func Normalize(sql string, redact string) (result string)

Normalize generates the normalized statements. it will get normalized form of statement text which removes general property of a statement but keeps specific property. possible values for 'redact' is "OFF", "ON" or "MARKER". Passing "" is seen as "OFF".

when "OFF", it is returned as is for example, when "ON": Normalize('select 1 from b where a = 1') => 'select ? from b where a = ?' for example, when "MARKER": Normalize('select 1 from b where a = 1') => 'select ‹1› from b where a = ‹1›'

func NormalizeForBinding 

func NormalizeForBinding(sql string, forPlanReplayerReload bool) (result string)

NormalizeForBinding generates the normalized statements with additional binding rules it will get normalized form of statement text which removes general property of a statement but keeps specific property.

for example: NormalizeForBinding('select 1 from b where a = 1') => 'select ? from b where a = ?'

func NormalizeKeepHint 

func NormalizeKeepHint(sql string) (result string)

NormalizeKeepHint generates the normalized statements, but keep the hints. it will get normalized form of statement text with hints. which removes general property of a statement but keeps specific property.

for example: Normalize('select /*+ use_index(t, primary) */ 1 from b where a = 1') => 'select /*+ use_index(t, primary) */ ? from b where a = ?'

func ParseErrorWith 

func ParseErrorWith(errstr string, lineno int) error

ParseErrorWith returns "You have a syntax error near..." error message compatible with mysql.

func ParseHint 

func ParseHint(input string, sqlMode mysql.SQLMode, initPos Pos) ([]*ast.TableOptimizerHint, []error)

ParseHint parses an optimizer hint (the interior of `/*+ ... */`).

func TrimComment 

func TrimComment(txt string) string

TrimComment trim comment for special comment code of MySQL.

Types

type CharsetClient 

type CharsetClient string

CharsetClient specifies the charset of a SQL. This is used to decode the SQL into a utf-8 string.

func (CharsetClient) ApplyOn 

func (c CharsetClient) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type CharsetConnection 

type CharsetConnection string

CharsetConnection is used for literals specified without a character set.

func (CharsetConnection) ApplyOn 

func (c CharsetConnection) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type CollationConnection 

type CollationConnection string

CollationConnection is used for literals specified without a collation.

func (CollationConnection) ApplyOn 

func (c CollationConnection) ApplyOn(p *Parser) error

ApplyOn implements ParseParam interface.

type Digest 

type Digest struct {
	// contains filtered or unexported fields
}

Digest stores the fixed length hash value.

func DigestHash deprecated 

func DigestHash(sql string) (digest *Digest)

DigestHash generates the digest of statements. it will generate a hash on normalized form of statement text which removes general property of a statement but keeps specific property.

for example: both DigestHash('select 1') and DigestHash('select 2') => e1c71d1661ae46e09b7aaec1c390957f0d6260410df4e4bc71b9c8d681021471

Deprecated: It is logically consistent with NormalizeDigest.

func DigestNormalized 

func DigestNormalized(normalized string) (digest *Digest)

DigestNormalized generates the digest of a normalized sql. it will generate a hash on a normalized sql. Normalize + DigestNormalized equals to NormalizeDigest.

for example: DigestNormalized('select ?') DigestNormalized should be called with a normalized SQL string (like 'select ?') generated by function Normalize. do not call with SQL which is not normalized, DigestNormalized('select 1') and DigestNormalized('select 2') is not the same

func NewDigest 

func NewDigest(b []byte) *Digest

NewDigest returns a new digest.

func NormalizeDigest 

func NormalizeDigest(sql string) (normalized string, digest *Digest)

NormalizeDigest combines Normalize and DigestNormalized into one method.

func NormalizeDigestForBinding 

func NormalizeDigestForBinding(sql string) (normalized string, digest *Digest)

NormalizeDigestForBinding combines Normalize and DigestNormalized into one method with additional binding rules.

func (*Digest) Bytes 

func (d *Digest) Bytes() []byte

Bytes returns the digest byte slice.

func (*Digest) String 

func (d *Digest) String() string

String returns the digest hex string.

type KeywordsType 

type KeywordsType struct {
	Word     string
	Reserved bool
	Section  string
}

KeywordsType defines the attributes of keywords

type ParseParam 

type ParseParam interface {
	ApplyOn(*Parser) error
}

ParseParam represents the parameter of parsing.

type Parser 

type Parser struct {
	// contains filtered or unexported fields
}

Parser represents a parser instance. Some temporary objects are stored in it to reduce object allocation during Parse function.

func New 

func New() *Parser

New returns a Parser object with default SQL mode.

func (*Parser) EnableWindowFunc 

func (parser *Parser) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the parser to parse syntax related with window function.

func (*Parser) Parse 

func (parser *Parser) Parse(sql, charset, collation string) (stmt []ast.StmtNode, warns []error, err error)

Parse parses a query string to raw ast.StmtNode. If charset or collation is "", default charset and collation will be used.

func (*Parser) ParseOneStmt 

func (parser *Parser) ParseOneStmt(sql, charset, collation string) (ast.StmtNode, error)

ParseOneStmt parses a query and returns an ast.StmtNode. The query must have one statement, otherwise ErrSyntax is returned.

func (*Parser) ParseSQL 

func (parser *Parser) ParseSQL(sql string, params ...ParseParam) (stmt []ast.StmtNode, warns []error, err error)

ParseSQL parses a query string to raw ast.StmtNode.

func (*Parser) Reset 

func (parser *Parser) Reset()

Reset resets the parser.

func (*Parser) SetParserConfig 

func (parser *Parser) SetParserConfig(config ParserConfig)

SetParserConfig sets the parser config.

func (*Parser) SetSQLMode 

func (parser *Parser) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for parser.

func (*Parser) SetStrictDoubleTypeCheck 

func (parser *Parser) SetStrictDoubleTypeCheck(val bool)

SetStrictDoubleTypeCheck enables/disables strict double type check.

type ParserConfig 

type ParserConfig struct {
	EnableWindowFunction        bool
	EnableStrictDoubleTypeCheck bool
	SkipPositionRecording       bool
}

ParserConfig is the parser config.

type Pos 

type Pos struct {
	Line   int
	Col    int
	Offset int
}

Pos represents the position of a token.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner implements the yyLexer interface.

func NewScanner 

func NewScanner(s string) *Scanner

NewScanner returns a new scanner object.

func (*Scanner) AppendError 

func (s *Scanner) AppendError(err error)

AppendError sets error into scanner. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) AppendWarn 

func (s *Scanner) AppendWarn(err error)

AppendWarn sets warning into scanner.

func (*Scanner) EnableWindowFunc 

func (s *Scanner) EnableWindowFunc(val bool)

EnableWindowFunc controls whether the scanner recognize the keywords of window function.

func (*Scanner) Errorf 

func (s *Scanner) Errorf(format string, a ...interface{}) (err error)

Errorf tells scanner something is wrong. Scanner satisfies yyLexer interface which need this function.

func (*Scanner) Errors 

func (s *Scanner) Errors() (warns []error, errs []error)

Errors returns the errors and warns during a scan.

func (*Scanner) GetSQLMode 

func (s *Scanner) GetSQLMode() mysql.SQLMode

GetSQLMode return the SQL mode of scanner.

func (*Scanner) InheritScanner 

func (s *Scanner) InheritScanner(sql string) *Scanner

InheritScanner returns a new scanner object which inherits configurations from the parent scanner.

func (*Scanner) Lex 

func (s *Scanner) Lex(v *yySymType) int

Lex returns a token and store the token value in v. Scanner satisfies yyLexer interface. 0 and invalid are special token id this function would return: return 0 tells parser that scanner meets EOF, return invalid tells parser that scanner meets illegal character.

func (*Scanner) LexLiteral 

func (s *Scanner) LexLiteral() interface{}

LexLiteral returns the value of the converted literal

func (*Scanner) SetSQLMode 

func (s *Scanner) SetSQLMode(mode mysql.SQLMode)

SetSQLMode sets the SQL mode for scanner.

Source Files

View all Source files

Directories

Path Synopsis
Package ast is the abstract syntax tree parsed from a SQL statement by parser.
 Package ast is the abstract syntax tree parsed from a SQL statement by parser.
Package duration provides a customized duration, which supports unit 'd', 'h' and 'm'
 Package duration provides a customized duration, which supports unit 'd', 'h' and 'm'
Goyacc is a version of yacc generating Go parsers.
 Goyacc is a version of yacc generating Go parsers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.