pos

Documentation

Overview

Package pos model data The data from https://github.com/fxsjy/jieba

Index

• type Dict
  • func (d *Dict) AddToken(text string, frequency float64, pos ...string)
  • func (d *Dict) Frequency(key string) (float64, bool)
  • func (d *Dict) Pos(key string) (string, bool)
• type SegPos
• type Segmenter
  • func (seg *Segmenter) Cut(sentence string, hmm bool) (result []SegPos)
  • func (seg *Segmenter) LoadDict(fileName ...string) error
  • func (seg *Segmenter) Trim(se []SegPos) (re []SegPos)
  • func (seg *Segmenter) TrimPos(se []SegPos, pos ...string) (re []SegPos)
  • func (seg *Segmenter) WithGse(segs gse.Segmenter)

Types

type Dict

type Dict struct {
	// contains filtered or unexported fields
}

Dict represents a dictionary used for word segmentation.

func (*Dict) AddToken

func (d *Dict) AddToken(text string, frequency float64, pos ...string)

AddToken adds one token

func (*Dict) Frequency

func (d *Dict) Frequency(key string) (float64, bool)

Frequency returns the frequency and existence of give word

func (*Dict) Pos

func (d *Dict) Pos(key string) (string, bool)

Pos returns the POS and existence of give word

type SegPos

type SegPos struct {
	Text, Pos string
}

SegPos represents a word with it's POS

type Segmenter

type Segmenter struct {
	// contains filtered or unexported fields
}

Segmenter is a words segmentation struct.

func (*Segmenter) Cut

func (seg *Segmenter) Cut(sentence string, hmm bool) (result []SegPos)

Cut cuts a sentence into words. Parameter hmm controls whether to use the HMM.

func (*Segmenter) LoadDict

func (seg *Segmenter) LoadDict(fileName ...string) error

LoadDict loads dictionary from given file name.

func (*Segmenter) Trim

func (seg *Segmenter) Trim(se []SegPos) (re []SegPos)

Trim not space and punct

func (*Segmenter) TrimPos

func (seg *Segmenter) TrimPos(se []SegPos, pos ...string) (re []SegPos)

TrimPos trim some pos

func (*Segmenter) WithGse

func (seg *Segmenter) WithGse(segs gse.Segmenter)

WithGse register gse segmenter