tokenizer

package

v0.1.0 Latest Latest Go to latest Published: Jul 17, 2014 License: Apache-2.0 Imports: 4 Imported by: 0

Details

Valid go.mod file

The Go module system was introduced in Go 1.11 and is the official dependency management solution for Go.
Redistributable license

Redistributable licenses place minimal restrictions on how software can be used, modified, and redistributed.
Tagged version

Modules with tagged versions give importers more predictable builds.
Stable version

When a project reaches major version v1 it is considered stable.
Learn more about best practices

View Source

const (
	BOSEOS = -1
)

This section is empty.

func NewLattice() *lattice

type Morph struct {
	Id         int
	Class      NodeClass
	Start, End int
	Surface    string
	// contains filtered or unexported fields
}

func (this Morph) Content() (content dic.Content, err error)

func (this Morph) String() string

type Node struct {
	// contains filtered or unexported fields
}

func (this Node) String() string

type NodeClass byte

const (
	KNOWN NodeClass = iota
	UNKNOWN
	INSERTED
	USER
	DUMMY
)

func (this NodeClass) String() string

type NodePool struct {
	// contains filtered or unexported fields
}

func NewNodePool(a_size int) *NodePool

type Tokenizer struct {
	// contains filtered or unexported fields
}

func NewTokenizer() *Tokenizer

func (this *Tokenizer) SetUserDic(a_udic *dic.UserDic)

func (this *Tokenizer) Tokenize(a_input string) (morphs []Morph, err error)