textkit

README

textkit

A tokeniser written in Go

Documentation

Overview

Package textkit provides a tokeniser and morphological analyser.

Index

• type LexicalEntry
• type Location
  • func (loc Location) String() string
• type MorphologicalLexicon
  • func NewMorphologicalLexicon(filename string) (*MorphologicalLexicon, error)
  • func (lex *MorphologicalLexicon) AddEntry(form, lemma, tag string)
  • func (lex *MorphologicalLexicon) Analyse(form string) []*LexicalEntry
• type Token
  • func (t *Token) String() string
• type TokenType
  • func (t TokenType) String() string
• type Tokeniser
  • func (t *Tokeniser) Tokenise(text, file string) []*Token

Types

type LexicalEntry

type LexicalEntry struct {
	// The entry's lemma.
	Lemma string
	// The morphological tag.
	Tag string
}

LexicalEntry is a lexical entry.

type Location

type Location struct {
	File   string
	Line   int
	Column int
}

Location specifies the location of a token.

func (Location) String

func (loc Location) String() string

type MorphologicalLexicon

type MorphologicalLexicon struct {
	// contains filtered or unexported fields
}

MorphologicalLexicon is a morphological lexicon.

func NewMorphologicalLexicon

func NewMorphologicalLexicon(filename string) (*MorphologicalLexicon, error)

NewMorphologicalLexicon returns a new morphological lexicon.

func (*MorphologicalLexicon) AddEntry

func (lex *MorphologicalLexicon) AddEntry(form, lemma, tag string)

AddEntry adds an entry to the lexicon.

func (*MorphologicalLexicon) Analyse

func (lex *MorphologicalLexicon) Analyse(form string) []*LexicalEntry

Analyse analyses a word form. Returns a list of lexical entries.

type Token

type Token struct {
	// The token's type.
	Type TokenType
	// The form of the token as a slice of runes.
	Form []rune
	// The location where the token is located.
	Loc Location
	// An associated tag.
	Tag string
}

Token is a text token.

func (*Token) String

func (t *Token) String() string

type TokenType

type TokenType int

TokenType is a token type.

const (
	Word TokenType = iota
	Number
	String
	Symbol
	EOL
	EndIndent
	EOF
)

Token types

func (TokenType) String

func (t TokenType) String() string

type Tokeniser

type Tokeniser struct {
	CommentPrefix  string
	StringRune     rune
	IdentChars     string
	KeepEOLs       bool
	KeepEndIndents bool
}

Tokeniser is a tokeniser which takes into account comments and special characters in identifiers.

func (*Tokeniser) Tokenise

func (t *Tokeniser) Tokenise(text, file string) []*Token

Tokenise tokenises a text.