Help us shape the future of Go! Take the 2026 Go Developer Survey

Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
- API
  
  Reference documentation for the pkg.go.dev API
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

gotoken

package module

Go to main page

Versions in this module

v0

v0.1.0

Aug 7, 2026

Changes in this version

+ type CountPolicy struct

+ func NewCountPolicy(maxCount int) *CountPolicy

+ func (p *CountPolicy) Depth(segments int) int

+ type FixedPolicy struct

+ func NewFixedPolicy(depth int) *FixedPolicy

+ func (p *FixedPolicy) Depth(segments int) int

+ type LinearPolicy struct

+ func NewLinearPolicy(shortLen, shortDepth, longLen, longDepth int) *LinearPolicy

+ func (p *LinearPolicy) Depth(segments int) int

+ type Option func(*Tokenizer)

+ func WithLanguage(rt *unicode.RangeTable) Option

+ func WithLanguages(tables ...*unicode.RangeTable) Option

+ func WithPolicy(p Policy) Option

+ type Policy interface

+ Depth func(segments int) int

+ type RuneClass int

+ const ClassDigit

+ const ClassLetter

+ const ClassOther

+ const ClassPunct

+ type TokenInfo struct

+ Base [2]int

+ Language int

+ type Tokenizer struct

+ func New(opts ...Option) *Tokenizer

+ func (t *Tokenizer) Tokenize(source string) map[string]TokenInfo