snowball

README

Go Target for Snowball

The initial implementation was built as a port of the Rust target. The initial focus has been on getting it to function, and making it work correctly. No attempt has been made to beautify the implementation, generated code, or address performance issues.

Usage

To generate Go source for a Snowball algorithm:

$ snowball path/to/algorithm.sbl -go -o algorithm

Go specific options

-Package/-P: the package name used in the generated go file (defaults to snowball)

-goruntime/-gor: the import path used for the Go Snowball runtime (defaults to github.com/snowballstem/snowball/go)

Snowball 3.0.1 and earlier supported -go/-gopackage, but this was just an alias for -Package/-P since Snowball 2.0.0.

Code Organization

compiler/generator_go.c has the Go code generation logic

go/ contains the default Go Snowball runtime support

go/stemwords contains the source for a Go version of the stemwords utility

go/algorithms location where the makefile generated code will end up

Using the Generated Stemmers

Assuming you generated a stemmer, put that code in a package which is imported by this code as english.

env := snowball.NewEnv("")
env.SetCurrent("beautiful")
english.Stem(env)
fmt.Printf("stemmed word is: %s", env.Current())

If you are stemming many words you should reuse env as shown above. If you are stemming a single word, you can set the current string when you create it with env:= snowball.NewEnv("beautiful"), but doing this when stemming many words is not recommended as the overhead is measurable (stemwords for the arabic stemmer on the sample vocabulary is about 12% faster if you reuse env).

Testing

Only the existing Snowball algorithms have been used for testing. This does not exercise all features of the language.

Run:

$ make check_go

An initial pass of fuzz-testing has been performed on the generated stemmers for the algorithms in this repo. Each ran for 5 minutes and used an initial corpus seeded with 10k words from the algorithm's snowballstem-data voc.txt file.

Known Limitations

• Code going through generate_dollar production has not been tested
• Code going through generate_debug production has not been tested

Documentation

Index

• Constants
• func RuneCountInString(str string) int
• type Among
  • func (a *Among) String() string
• type AmongF
• type Env
  • func NewEnv(val string) *Env
  • func (env *Env) AssignTo() string
  • func (env *Env) Clone() *Env
  • func (env *Env) Current() string
  • func (env *Env) Debug(count, lineNumber int)
  • func (env *Env) EqS(s string) bool
  • func (env *Env) EqSB(s string) bool
  • func (env *Env) FindAmong(amongs []*Among, ctx interface{}) int32
  • func (env *Env) FindAmongB(amongs []*Among, ctx interface{}) int32
  • func (env *Env) GoInGrouping(chars []byte, min, max int32) bool
  • func (env *Env) GoInGroupingB(chars []byte, min, max int32) bool
  • func (env *Env) GoOutGrouping(chars []byte, min, max int32) bool
  • func (env *Env) GoOutGroupingB(chars []byte, min, max int32) bool
  • func (env *Env) Hop(delta int) bool
  • func (env *Env) HopBack(delta int) bool
  • func (env *Env) HopBackChecked(delta int) bool
  • func (env *Env) HopChecked(delta int) bool
  • func (env *Env) InGrouping(chars []byte, min, max int32) bool
  • func (env *Env) InGroupingB(chars []byte, min, max int32) bool
  • func (env *Env) Insert(bra, ket int, s string)
  • func (env *Env) NextChar()
  • func (env *Env) OutGrouping(chars []byte, min, max int32) bool
  • func (env *Env) OutGroupingB(chars []byte, min, max int32) bool
  • func (env *Env) PrevChar()
  • func (env *Env) ReplaceS(bra, ket int, s string) int32
  • func (env *Env) SetCurrent(s string)
  • func (env *Env) SliceDel()
  • func (env *Env) SliceFrom(s string)
  • func (env *Env) SliceTo() string

Constants

const MaxInt = math.MaxInt32

const MinInt = math.MinInt32

Functions

func RuneCountInString

func RuneCountInString(str string) int

RuneCountInString is a wrapper around utf8.RuneCountInString this allows us to not have to conditionally include the utf8 package into some stemmers and not others

Types

type Among

type Among struct {
	Str string
	A   int32
	B   int32
	F   AmongF
}

func (*Among) String

func (a *Among) String() string

type AmongF

type AmongF func(env *Env, ctx interface{}) bool

type Env

type Env struct {
	Cursor        int
	Limit         int
	LimitBackward int
	Bra           int
	Ket           int
	// contains filtered or unexported fields
}

Env represents the Snowball execution environment

func NewEnv

func NewEnv(val string) *Env

NewEnv creates a new Snowball execution environment on the provided string

func (*Env) AssignTo

func (env *Env) AssignTo() string

func (*Env) Clone

func (env *Env) Clone() *Env

func (*Env) Current

func (env *Env) Current() string

func (*Env) Debug

func (env *Env) Debug(count, lineNumber int)

func (*Env) EqS

func (env *Env) EqS(s string) bool

func (*Env) EqSB

func (env *Env) EqSB(s string) bool

func (*Env) FindAmong

func (env *Env) FindAmong(amongs []*Among, ctx interface{}) int32

func (*Env) FindAmongB

func (env *Env) FindAmongB(amongs []*Among, ctx interface{}) int32

func (*Env) GoInGrouping

func (env *Env) GoInGrouping(chars []byte, min, max int32) bool

func (*Env) GoInGroupingB

func (env *Env) GoInGroupingB(chars []byte, min, max int32) bool

func (*Env) GoOutGrouping

func (env *Env) GoOutGrouping(chars []byte, min, max int32) bool

func (*Env) GoOutGroupingB

func (env *Env) GoOutGroupingB(chars []byte, min, max int32) bool

func (*Env) Hop

func (env *Env) Hop(delta int) bool

func (*Env) HopBack

func (env *Env) HopBack(delta int) bool

func (*Env) HopBackChecked

func (env *Env) HopBackChecked(delta int) bool

func (*Env) HopChecked

func (env *Env) HopChecked(delta int) bool

func (*Env) InGrouping

func (env *Env) InGrouping(chars []byte, min, max int32) bool

func (*Env) InGroupingB

func (env *Env) InGroupingB(chars []byte, min, max int32) bool

func (*Env) Insert

func (env *Env) Insert(bra, ket int, s string)

func (*Env) NextChar

func (env *Env) NextChar()

func (*Env) OutGrouping

func (env *Env) OutGrouping(chars []byte, min, max int32) bool

func (*Env) OutGroupingB

func (env *Env) OutGroupingB(chars []byte, min, max int32) bool

func (*Env) PrevChar

func (env *Env) PrevChar()

func (*Env) ReplaceS

func (env *Env) ReplaceS(bra, ket int, s string) int32

func (*Env) SetCurrent

func (env *Env) SetCurrent(s string)

func (*Env) SliceDel

func (env *Env) SliceDel()

func (*Env) SliceFrom

func (env *Env) SliceFrom(s string)

func (*Env) SliceTo

func (env *Env) SliceTo() string