tokenizer

README

tokenizer

tokenizer is a custom token decomposer.

Documentation

Index

• Constants
• type Parser
• type Token
• type TokenKind
• type TokenPattern
• type TokenSet
• type TokenType

Constants

const (
	KindKeyWord = iota
	KindIdentify
	KindDigits
	KindOperator
	KindSplit
)

Types

type Parser

type Parser struct {
	Path   string
	Buffer []byte
	P1     int
	P2     int
	Line   int
	Col    int
}

type Token

type Token struct {
	Path  string
	Type  TokenType
	Value string
	Line  int
	Col   int
}

type TokenKind

type TokenKind int

type TokenPattern

type TokenPattern struct {
}

type TokenSet

type TokenSet struct {
}

TokenSet to predefine token patterns

type TokenType

type TokenType int